robots.txt validator and checker: find syntax errors and test bot rules
A robots.txt validator reads your file line by line and reports every line crawlers cannot use: a missing colon, a misspelled field, a rule outside a group, a relative path. Paste your file or enter a site address; this free tool lists each problem with its line number and then shows which AI bots your rules block. Every check is backed by RFC 9309 or Google's robots.txt documentation.
We fetch /robots.txt of that site, nothing else.
How to check your robots.txt
- Enter your site address to fetch
/robots.txt, or switch to Paste robots.txt to test a draft before you publish it. - Read the counts. Errors are lines that break a rule or the whole file, warnings are lines crawlers ignore, notes are things worth a second look.
- Fix each finding in the line it names. The finding code links to its explanation below.
- Check the bot table. It answers a different question than syntax: whether each bot may fetch
/under your rules. For one specific page, use the AI crawler checker. - Every URL in your sitemap can be tested the same way with the sitemap robots.txt check. To write a file from scratch, use the robots.txt generator.
What each check means
The validator reports the codes below. A finding appears only when RFC 9309 or Google's robots.txt specification supports it. Counts and severities are ours: an error breaks the file, a warning is a line that does nothing, a note is context.
HTML page instead of robots.txt Error html-instead-of-robots
The file starts with HTML markup. Many servers and single-page apps answer /robots.txt with their home or error page. Crawlers read it as robots.txt, find no valid rules and treat the site as unrestricted.
Fix: Serve a plain-text file at /robots.txt with the content type text/plain, or remove the catch-all rewrite for that path.
Not a text file Error not-text
The response is binary or compressed data, for example a gzip body the server sent without a matching Content-Encoding header. Crawlers cannot read any rules from it, so the file has no effect.
Fix: Serve /robots.txt as plain UTF-8 text with the content type text/plain, and only compress it when the Content-Encoding header says so.
Line without a colon Warning missing-colon
Every rule is a field name, a colon and a value. A line without the colon, such as "Disallow /admin", is not a rule, so it is ignored. Google ignores invalid lines.
Fix: Write the line as "Field: value" with the colon.
Unknown or misspelled field Warning unknown-field
The field name is not one the linter knows, for example "Dissallow" or "Useragent". Crawlers skip lines they do not understand, so a misspelled Disallow rule does nothing. The message suggests the closest valid field when there is one.
Fix: Correct the spelling. The standard fields are User-agent, Allow, Disallow and Sitemap.
Rule before any User-agent line Warning rule-before-user-agent
Allow and Disallow lines belong to a group that starts with at least one User-agent line. A rule above the first User-agent line applies to no crawler.
Fix: Move the rule below a User-agent line, or add "User-agent: *" above it.
Path does not start with / or * Warning path-not-absolute
Rule paths are matched against the URL path, which always starts with a slash. A value such as "admin" or a full URL never matches a path.
Fix: Start the path with "/" (or "*" for a wildcard), for example "Disallow: /admin/".
Sitemap is not an absolute URL Warning sitemap-not-absolute
The Sitemap field takes the full address of a sitemap, including http:// or https://. A relative path such as /sitemap.xml is not valid there.
Fix: Use the full URL, for example "Sitemap: https://example.com/sitemap.xml".
Directive Google does not support Note unsupported-directive
Crawl-delay, Noindex, Host and Clean-param are not part of RFC 9309. Google's documentation lists only user-agent, allow, disallow and sitemap, and says other fields such as crawl-delay are not supported. Some other crawlers honour Crawl-delay.
Fix: Keep the line only if a crawler you care about documents support for it. To keep a page out of Google, use a noindex robots meta tag or X-Robots-Tag header on a page that crawlers may fetch.
Extension outside RFC 9309 Note extension-directive
Content-Signal (Cloudflare's Content Signals Policy, with the signals search, ai-input and ai-train) and License (RSL, which points to a license.xml) are extensions in real use. They are not part of RFC 9309, so a crawler that does not support them skips the line; they do not allow or block anything by themselves.
Fix: Nothing to fix if you use them on purpose. Access is still decided by Allow and Disallow, so keep those rules explicit.
Source: Cloudflare: Content Signals Policy (24 September 2025), RSL standard
User-agent group without rules Note empty-group
The group has no Allow or Disallow line, so it allows everything. Because a crawler with its own group does not use the "*" group, such a group can make a bot more permissive than the rest of the file suggests (RFC 9309 section 2.2.1).
Fix: Add the rules the group should have, or delete the group if the bot should follow the "*" rules.
Source: RFC 9309
User-agent in more than one group Note duplicate-agent
The same user-agent name appears in separate groups. Crawlers merge the rules of all groups that name them, which is easy to get wrong when the groups were written at different times.
Fix: Combine the rules into one group so the file says what it means.
Unknown user-agent token Note unknown-agent
The name has the same letters as a bot we document with different punctuation (PerplexityUser for Perplexity-User), or one missing, doubled or swapped letter, or it is a legacy Anthropic token that is not on Anthropic's current list. A misspelled token matches no crawler, so its rules never apply.
Fix: Check the spelling against the operator's documentation or our bot directory.
Source: RFC 9309
Every AI crawler is blocked Warning all-ai-blocked
Each AI training, search and user-request crawler in our default list is blocked from "/". That also removes the site from AI search results. It is a valid choice, but often a side effect of a "User-agent: *" rule meant for something else.
Fix: If you want to appear in AI search, allow the search and user-request bots explicitly and block only training bots.
Source: RFC 9309
File larger than 500 KiB Warning too-large
Google reads only the first 500 KiB of a robots.txt file and ignores the rest, so rules at the end may never apply.
Fix: Shorten the file: merge rules, use wildcards and drop rules for URLs that no longer exist.
More than 20,000 lines Note too-many-lines
This tool evaluates only the first 20,000 lines of a file.
Fix: Shorten the file.
More findings than shown Note findings-truncated
At most 200 findings are listed; the count says how many more were found.
Fix: Fix the listed findings and run the check again.
Google's robots.txt report in Search Console lists the robots.txt files Google found for a property, when each was last fetched and any parsing issues. Last reviewed 2026-10-18.
Run the same check in CI
The checks come from the open-source crawlerpeek package, so a pipeline can fail when a robots.txt change introduces an error. The command exits with 1 on findings at or above --fail-on (default warning). The MCP server offers the same check as the tool lint_robots_txt.
npx crawlerpeek lint https://example.com npx crawlerpeek lint ./public/robots.txt --fail-on error
Questions
Does Google support Crawl-delay in robots.txt?
No. Google's robots.txt documentation lists four supported fields, user-agent, allow, disallow and sitemap, and says other fields such as crawl-delay aren't supported. Some other crawlers honour Crawl-delay, so the validator reports it as a note, not as an error.
Why does my robots.txt file have format errors?
A format error means a line that crawlers cannot read as a rule. The usual causes are a missing colon, a misspelled field name such as Dissallow, a rule above the first User-agent line, a path that does not start with /, a relative Sitemap address, or an HTML page served at /robots.txt. Google ignores invalid lines. The validator lists each one with its line number. Google's own robots.txt report in Search Console shows the fetch status, size and parsing issues of the files it found.
Do typos in robots.txt break the whole file?
No, but they silently remove the rule you meant. A line with a misspelled field is skipped and the other lines still apply. Two typos matter most: a misspelled Disallow leaves its group without rules, which allows everything, and a misspelled bot name matches no crawler, so its rules never apply. The validator suggests the closest valid field or bot token.
How do the wildcards * and $ work in robots.txt?
Per Google's documentation, * stands for zero or more of any characters and $ marks the end of the URL. Disallow: /*.pdf$ blocks every URL that ends in .pdf; Disallow: /private* is the same as Disallow: /private, because a trailing * is ignored. When several rules match, the one with the longest path wins and Allow wins a tie (RFC 9309, section 2.2.2).
How can I test whether Googlebot may crawl a URL?
The table below the result shows whether Googlebot, Bingbot and the AI bots may fetch the home page. For another page, run the AI crawler checker with that page's address; it applies the same rules to its path. Google also documents the URL Inspection tool in Search Console and an open-source robots.txt parser for testing a file locally.
Is the text I paste stored?
No. Pasted text is checked in memory on our server and discarded; it is not written to a URL, a log or a database. Only a rate-limit counter tied to a hash of your IP address is kept, for a short time.
More on the format: robots.txt syntax explained and common robots.txt mistakes. What 151 large sites put in their files: who blocks AI crawlers.