Crawlerpeek
Check a site

robots.txt explained: syntax, matching rules and where the file goes

robots.txt is a plain text file at the root of a host that tells crawlers which paths they may fetch. A crawler uses only the one group that names it most specifically, and within that group the most specific (longest) matching rule wins. It controls crawling, not indexing.

By the Crawlerpeek editorial team · Updated · How we check

Where the file goes

The file must be at the top level of the host: https://example.com/robots.txt. Crawlers do not look in subdirectories. Rules apply only to the exact host, protocol and port that serve the file, so blog.example.com needs its own robots.txt. Source: Google's robots.txt specification.

The four fields

FieldMeaningExample
User-agentStarts a group and names the crawler it applies toUser-agent: GPTBot
DisallowA path the crawler may not fetchDisallow: /private/
AllowA path the crawler may fetch (use to make an exception)Allow: /private/press/
SitemapThe full URL of a sitemapSitemap: https://example.com/sitemap.xml

Field names are not case-sensitive, but path values are, and they must start with a slash. Other fields such as crawl-delay are not supported by Google, although some other crawlers honor them. See the individual bot pages for what each operator says.

How a crawler picks its rules

  • Each crawler uses only one group: the one with the most specific user agent that matches it. All other groups are ignored. That is why a group named GPTBot replaces, rather than adds to, the User-agent: * group for that bot.
  • Within the group, when several rules match a URL, the most specific rule (the longest path) is used. When an Allow and a Disallow tie, Google uses the least restrictive rule, that is Allow.
  • * matches zero or more characters; $ marks the end of the URL. Disallow: /*.pdf$ blocks URLs that end in .pdf.
User-agent: *
Disallow: /private/
Allow: /private/press/

User-agent: GPTBot
Disallow: /

In this file GPTBot is blocked from everything. A crawler without its own group, such as OAI-SearchBot, follows the * group and may fetch everything except /private/, apart from /private/press/.

Limits and error handling

  • Google reads at most 500 KiB of the file; content beyond that is ignored.
  • A 4xx answer (except 429) is treated as if no robots.txt exists, so nothing is restricted.
  • A 5xx answer makes Google stop crawling the site for the first 12 hours while it retries, then use the last good cached copy for up to 30 days.
  • Google generally caches robots.txt for up to 24 hours, so changes are not instant. Other operators give their own delays on their bot pages (for example 24 hours for Meta and Amazon, 72 hours for DuckDuckGo).

What to read next

Most mistakes come from treating robots.txt as a privacy or indexing tool: see robots.txt mistakes and how to test your file. To build a file for AI bots, use the generator.

See how your own site does.Run the checker

Keep reading