URL blocked by robots.txt: what the Search Console message means and how to fix it

A Disallow rule in your robots.txt stops Googlebot from crawling this URL. If the page should be in Google, remove or narrow that rule and validate the fix.

By the Crawlerpeek editorial team · Sources checked 2026-10-30 · Updated 2026-10-30 · Editorial standards

What Google says

This page was blocked by your site's robots.txt file. Note that this does not guarantee that the page won't be indexed through some other means.

Source: Google Search Console Help: Page indexing report, read 2026-10-30.

Common causes

• A rule that is broader than intended. Rules match from the start of the path, so Disallow: /p also blocks /products and /pricing; a Disallow: / copied from a staging site blocks everything.
• A wildcard that catches too much. In Google's syntax * matches any sequence of characters, so Disallow: /*? blocks every URL with a query string.
• A group that names Googlebot. Google uses only the group with the most specific user agent that matches, so once User-agent: Googlebot exists, the rules under User-agent: * no longer apply to Googlebot (Google's robots.txt specification).
• The URL is listed in your sitemap or linked internally although robots.txt closes it, so Google keeps reporting it.
• A deliberate block of admin, cart, account or internal search pages. That is the intended result, not an error.

How to check

1. Enter the exact URL in the AI crawler checker and read the Googlebot row: it applies your live robots.txt to that path with the longest-match rule.
2. Paste the file into the robots.txt validator to find syntax mistakes, such as a misspelled field, that change what a rule covers.
3. Test every URL in your sitemap at once with the sitemap robots.txt check; it lists, per bot, which URLs the rules block.
4. In Search Console, URL Inspection shows "Crawl allowed?". Google's help says a "No" there means Google can't crawl the page because of a robots.txt rule (URL Inspection tool).
5. After you change the file, the robots.txt report shows which version Google last fetched and offers "Request a recrawl". Google generally caches robots.txt for up to 24 hours.
6. Then click Validate fix in the Page indexing report. Google says validation typically takes up to about two weeks.

When to fix it and when to leave it

Leave it when the URL is one you meant to keep crawlers away from, such as checkout, account or internal search pages. If your goal is to keep a page out of Google, robots.txt is the wrong tool: Google says it "is not a mechanism for keeping a web page out of Google" (Introduction to robots.txt). Allow crawling and use noindex instead; see URL marked ‘noindex’. If a blocked URL shows up in Google anyway, that is Indexed, though blocked by robots.txt.

AI crawlers and AI search

Search Console reports Googlebot only. Rules for AI crawlers sit in their own groups: a User-agent: GPTBot group changes nothing for Googlebot, and blocking Google's Google-Extended token does not affect inclusion in Google Search (Google's common crawlers). The reverse matters more: Google says a page must be indexed and eligible to be shown with a snippet to appear as a supporting link in AI Overviews or AI Mode (AI features and your website), and a URL blocked for Googlebot is shown without a description at best. To see the decision for each AI bot side by side with Googlebot, run the checker.

Check the URL: robots.txt per bot, HTTP status, noindex and the HTML a crawler gets.Run the checker