We read the robots.txt of 151 big websites: who blocks AI crawlers, and what is broken

The short version

• 44 percent block AI. 59 of 135 readable files block at least one of GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot or OAI-SearchBot from /.
• ClaudeBot and Bytespider are blocked most (55 sites each, 41 percent), ahead of CCBot (53) and GPTBot (42).
• Training yes, search no. 22 sites block OpenAI's training crawler GPTBot but leave OAI-SearchBot, the crawler behind ChatGPT search, open. 20 block both.
• News blocks, software does not. 19 of 34 news sites block GPTBot; 2 of 33 software and SaaS sites do, and none of the developer-docs or SEO sites.
• A third carry dead lines. 44 files (33 percent) still name anthropic-ai or Claude-Web, which are not on Anthropic's current list of crawlers.
• The syntax is fine. In 135 files we found one line that silently does nothing. The bigger risk is the delivery: 9 sites answered /robots.txt with HTTP 403, 3 with a web page, and one with HTTP 418.

How we checked

We picked 151 well-known websites across nine groups: news, reference and community, shops, software and SaaS, developer docs, travel and marketplaces, media and social networks, health and education, and SEO and marketing. It is a hand-picked list of large sites, not a random sample of the web, so read the percentages as a picture of big publishers and platforms.

Each site got two requests to https://<domain>/robots.txt, one after the other with a 1.5-second pause, from an identifiable client (crawlerpeek/0.3, the open-source CLI). Redirects were followed, for example from example.com to www.example.com. The robots.txt validator logic checked syntax, and the same rules that crawlers use (RFC 9309, longest match wins) decided for each bot whether it may fetch the home page /. "Blocked" in this article always means: the file closes / to that bot. A site can still block a bot by other means, such as a firewall, and it can also serve a different robots.txt to a real crawler than to us.

Eight sites timed out on the first pass; we asked them once more and six answered. Everything here is what the files said on 29 October 2026.

Which AI crawlers are blocked

The pattern is clear: crawlers that collect training data are blocked two to three times as often as the crawlers that power AI search and fetch pages when a user asks. Many large sites have made the trade we describe in allow AI assistants, block training: keep the content out of training sets, stay visible where answers link back. The one site that closes / to Googlebot in the file we received is reddit.com, which also closes it to Bingbot; goodreads.com closes it to Bingbot.

Note the two control tokens. Google-Extended and Applebot-Extended never fetch anything; they only tell Google and Apple whether crawled pages may be used for their AI models. Blocking them does not remove a site from Google Search or Apple's search features, per Google and Apple.

Training yes, search no

OpenAI runs separate crawlers and documents a separate robots.txt token for each (OpenAI's crawler overview): GPTBot for training, OAI-SearchBot for ChatGPT search, ChatGPT-User for fetches a user triggers. Of the 42 sites that block GPTBot, 22 leave OAI-SearchBot open and 20 block both. The 20 that block both include nytimes.com, bbc.com, economist.com, reddit.com, amazon.com, and the large social networks.

Anthropic made the same split: ClaudeBot for training, Claude-SearchBot for search, Claude-User for user requests (Anthropic's crawler page). ClaudeBot is blocked on 55 sites, Claude-SearchBot on 30. If you want the same trade-off, our robots.txt generator writes it for you.

News blocks, software does not

Publishers whose product is the text itself block most. Software companies, whose documentation is how AI assistants learn to recommend and explain their products, almost never do. For a company that wants to be named in AI answers, closing the door to the crawlers behind those answers works against that goal.

13 sites close the door to every AI crawler

On 13 sites every AI training, search and user crawler in our default list is blocked from /: nytimes.com, usatoday.com, cnbc.com, theconversation.com, reddit.com, imdb.com, amazon.com, yelp.com, x.com, facebook.com, instagram.com, pinterest.com and tiktok.com. Several of them do it with an allow list: User-agent: * with Disallow: /, then named groups for the few crawlers they let in. That is the strictest pattern, and it also shuts out every new AI crawler by default, including the search and assistant fetchers that would send visitors back.

A third still name tokens Anthropic no longer lists

44 files (33 percent) contain anthropic-ai (42 files) or Claude-Web (31 files). Neither is on Anthropic's current list, which names ClaudeBot, Claude-User and Claude-SearchBot. In 43 of the 44 files ClaudeBot is named as well, so the old lines are dead weight rather than a hole. The exception is cloudflare.com: it allows Anthropic-AI and Claude-Web by name but does not name ClaudeBot, which therefore falls under the file's * group.

Dead lines are harmless for crawlers but not for the people who maintain the file: they suggest a rule exists for Anthropic's crawlers when the rule that applies is a different one. Our linter reports them as unknown-agent notes.

Mistakes we found in the syntax

Almost none. Across 135 files, one line does nothing because of a syntax error: line 564 of healthline.com's file reads Disallow /account/* without the colon, so crawlers skip it. Two near-misses turned out to be harmless: yelp.com lists PerplexityUser next to the correct Perplexity-User, and elpais.com lists Amazon-Bot next to Amazonbot. Large sites clearly test their files.

Smaller things were common: 23 files (17 percent) use Crawl-delay, which Google does not support (some other crawlers do), one uses Noindex, which no major crawler reads in robots.txt, and 20 name the same user agent in more than one group. Crawlers merge such groups (RFC 9309 section 2.2.1), but it makes a file easy to misread.

New signals: Content-Signal and RSL

Nine files carry a Content-Signal line, Cloudflare's Content Signals Policy from September 2025, which states preferences for search, ai-input and ai-train: cloudflare.com, corriere.it, heroku.com, intercom.com, neilpatel.com, netlify.com, sentry.io, theatlantic.com and twilio.com. Two, medium.com and theguardian.com, carry a License line from the RSL standard, which points to machine-readable licensing terms. Neither is part of RFC 9309. A crawler that does not support them skips the line; access is still decided by Allow and Disallow.

When the robots.txt itself is blocked

16 of 151 sites gave us no readable file. Nine answered with HTTP 403: cnn.com, politico.com, homedepot.com, adidas.com, npmjs.com, mayoclinic.org, nih.gov, udemy.com and searchengineland.com. readthedocs.io, linkedin.com and khanacademy.org answered with a web page instead of a text file, stackoverflow.com with HTTP 418, nextjs.org with HTTP 404, and two did not answer in time or failed on a redirect.

Bot protection is the likely cause: our client is not a browser, so it gets the treatment an unknown bot gets. That matters, because under RFC 9309 a 4xx answer for robots.txt means a crawler may fetch everything, and Google treats it the same way (Google's robots.txt specification). A firewall that blocks /robots.txt for a new AI crawler does not keep it out by robots.txt; it hides the rules from it. Whether the real crawlers get a different answer we cannot see from the outside. If you run bot protection, allow /robots.txt for every client and check it with the validator.

Compressed although nobody asked

One site, www.forbes.com, sends its robots.txt gzip-compressed even when the request asks for no compression. Real crawlers decode that. Our own CLI did not until this study: before version 0.3.1 it read the file as binary data and reported no rules at all. We fixed it the same day (changelog). If you test robots.txt with your own scripts, make sure they decode Content-Encoding.

Check your own file

• Paste your robots.txt or enter your domain in the robots.txt validator: syntax, legacy tokens, extensions, and a table of which AI crawlers may fetch your home page.
• Check every URL in your sitemap against your robots.txt with the sitemap robots.txt check, to find pages that are blocked by accident.
• Run the same checks in CI: npx crawlerpeek lint https://example.com fails the build on warnings, npx crawlerpeek scan https://example.com checks the sitemap. See CLI and MCP.
• To choose which bots to let in, start from the robots.txt generator or the AI crawler directory.