Indexed, though blocked by robots.txt: why Google lists the page and which fix to pick
Google indexed the URL from links without crawling it, because robots.txt blocks it. To remove it, allow crawling and add noindex; to keep it, lift the block.
The message in Search Console
Indexed, though blocked by robots.txt
Page indexing report, group "Indexed, with a warning". Spelling as in Google's help page for the report.
By the Crawlerpeek editorial team · Sources checked · Updated · Editorial standards
What Google says
The page was indexed despite being blocked by your website's robots.txt file. Google always respects robots.txt, but this doesn't necessarily prevent indexing if someone else links to your page. Because of the robots.txt rule, any snippet shown in Google Search results for the page will probably be very limited.
Source: Google Search Console Help: Page indexing report, read 2026-10-30.
Common causes
- Other pages, on your site or elsewhere, link to the blocked URL. Google can index a URL from those links without fetching it.
- The URL was blocked to hide it from search. robots.txt only controls crawling, so this is the expected side effect.
- A rule blocks more than intended, for example a prefix or wildcard that also matches pages you want in Google. See URL blocked by robots.txt for the usual patterns.
- A noindex was added to a page that robots.txt blocks. Google says the crawler will never see the noindex rule then (Block Search indexing with noindex).
How to check
- Enter the URL in the AI crawler checker: the Googlebot row shows whether your current robots.txt still blocks it, and the HTML section shows whether the page carries noindex.
- Paste the file into the robots.txt validator to check its syntax line by line, then look for the Disallow rule whose path matches the start of the blocked URL.
- In URL Inspection, "Crawl allowed?" shows "No" for a robots.txt block. Google notes that "Indexing allowed?" then always shows "Yes", because Google cannot see a noindex rule on a page it may not crawl (URL Inspection tool).
When to fix it and when to leave it
Decide what you want first. If the page should be in Google, delete or narrow the Disallow rule and let Google crawl it. If the page should not be in Google, Google's advice is to remove the robots.txt block and use noindex, in a robots meta tag or an X-Robots-Tag header; the page must stay crawlable until Google has seen it. Google says it can take months to revisit a page, depending on its importance, and that you can ask for a recrawl with URL Inspection. Leaving the warning alone is a choice too: the URL stays listed, with a very limited snippet.
AI crawlers and AI search
Google says a page must be indexed and eligible to be shown with a snippet to be a supporting link in AI Overviews or AI Mode (AI features and your website), and Google expects the snippet of a blocked page to be very limited. Blocking Google's Google-Extended token is a separate decision about Gemini training and grounding; Google says it does not affect inclusion in Google Search (Google's common crawlers).
Questions
Why does Google index a page that robots.txt blocks?
Google respects the block and does not crawl the page, but it can still index the URL using information from pages that link to it. robots.txt controls crawling, not indexing.
Should I add noindex to a page blocked by robots.txt?
Only after you remove the block. A crawler that may not fetch the page never sees the noindex rule, so the combination does not work.
Is this warning bad for my site?
It is a warning, not an error. Google says warnings don't prevent a page from being indexed, but they do reduce Google's ability to understand and index your pages. Fix it if the URL should either rank normally or disappear from Google.
Related messages
- URL blocked by robots.txt · A Disallow rule in your robots.txt stops Googlebot from crawling this URL. If the page should be in Google, remove or narrow that rule and validate the fix.
- URL marked ‘noindex’ · Google saw a noindex rule, in a robots meta tag or an X-Robots-Tag header, and left the page out. Remove the rule if the page should be in Google.
- Search Console Crawl and Indexing Errors
Sources
Google's documentation, each page read on 2026-10-30. Google updates these pages; if its wording has changed since, Google's page is the reference.
- Google Search Console Help: Page indexing report (checked 2026-10-30)
- Google Search Central: Block Search indexing with noindex (checked 2026-10-30)
- Google Search Console Help: URL Inspection tool (checked 2026-10-30)
- Google Search Central: AI features and your website (checked 2026-10-30)
- Google Search Central: Google's common crawlers (checked 2026-10-30)
Check the URL: robots.txt per bot, HTTP status, noindex and the HTML a crawler gets.Run the checker