Sitemap robots.txt check: which sitemap URLs are blocked?
This tool checks every URL in your sitemap against your robots.txt and shows, for each of 15 crawlers from Googlebot to GPTBot, how many URLs are blocked and which. It requests only robots.txt and the sitemap files, never the pages, and checks up to 100 URLs per run.
How the check works
- Sitemap discovery. The tool reads the
Sitemap:lines in robots.txt (up to five) and falls back to/sitemap.xml. A sitemap index is followed one level, up to five child sitemaps. - Same host only. URLs on another host are ignored, and so are sitemaps that are not on the checked host or the host its robots.txt redirects to.
- Rules. For each bot, the group that names it applies, otherwise the
*group. The longest matching rule wins and Allow wins a tie, as in RFC 9309. Wildcards and$work as in Google's specification. - Limits. Sitemap files up to 5 MB or 50,000 entries, no
.gzfiles, a few seconds per request. If a file cannot be read, the table says which and why.
Run the same check in CI
The crawlerpeek package runs this scan from a terminal or a pipeline, up to 200 URLs, and exits with 1 when --fail-on blocked finds a blocked URL. The MCP tool is scan_site.
npx crawlerpeek scan https://example.com npx crawlerpeek scan https://example.com --limit 100 --fail-on blocked
Questions
How do I check my sitemap URLs against robots.txt?
Enter your site address and a limit. The tool reads robots.txt and the sitemaps it lists (or /sitemap.xml), takes the first URLs from them and evaluates each one against the rules for 15 crawlers. The table shows how many are blocked per bot, and the lists below it show which.
Which requests does the tool make?
Only robots.txt and sitemap files: at most five sitemaps named in robots.txt and five child sitemaps of a sitemap index, each up to 5 MB or 50,000 entries. It never fetches the pages themselves, and it skips sitemap files on another host and .gz files.
Why would a sitemap URL be blocked?
A sitemap lists the URLs you want found; a Disallow rule that matches one contradicts that. It usually comes from a broad rule such as Disallow: /*? or Disallow: /blog/ written for another purpose. Which one matched is easiest to see by pasting the file into the robots.txt validator.
Why is only part of my sitemap checked?
The web tool evaluates up to 100 URLs, in the order the sitemap lists them, so a run stays quick and polite. The command line tool takes up to 200: npx crawlerpeek scan https://example.com --limit 200.
Does blocked in this table mean the page is not in Google?
No. The table applies robots.txt rules, which control crawling. Whether a URL is indexed is a separate question that depends on Google, not on this file. A blocked URL is one that the bot may not fetch.
Check the file itself with the robots.txt validator, or one page with the AI crawler checker. Last reviewed 2026-10-18.