robots.txt says no, your CDN says yes: do big sites enforce their AI crawler blocks?

The short answer

No, a robots.txt rule does not make a server refuse anything. It is a request that crawlers may or may not follow. What a site actually sends to a request with GPTBot in its user agent is decided by the web server, CDN or firewall in front of it, and nothing forces those rules to match the file. In our test the gap was wide: for every bot, about a third of the sites that disallow it still returned the page.

What we measured

From our robots.txt study of 151 large sites (files read on 29 October 2026) we took every site whose robots.txt disallows at least one of GPTBot (42 sites), ClaudeBot (55) or CCBot (53) from the home page: 59 sites in total. On 2 November 2026 we requested each site's home page once with a normal Chrome user agent, and once with the user agent string of each of those bots that the site disallows. That makes 209 requests. We recorded the status code, the final URL after redirects, the bytes received, and whether the answer was a challenge page (Cloudflare, DataDome, HUMAN/PerimeterX, Akamai, Imperva and AWS WAF markers).

We counted a request as **served** when it returned HTTP 200 with no challenge marker. Requests were sequential, at least 1.5 seconds apart, with a 15 second timeout, one retry only after a network error, and only public home pages. The script is ai-crawler-edge-scan.py and runs on the Python standard library alone; the per-request results are in the CSV linked below.

The numbers

Across the three bots, 23 of the 59 sites (39 percent) served the page to at least one crawler name they disallow, and 18 served it to every one they disallow. Twelve sites answered the Chrome request with the page and refused every crawler name: theguardian.com, theverge.com, vox.com, usatoday.com, cnbc.com, spiegel.de, corriere.it, nzz.ch, reddit.com, airbnb.com, x.com and tumblr.com. For those, robots.txt and the edge say the same thing.

Three more details from the data:

• **Some refusals are bot-specific, some are not.** Our plain Chrome request, which has no browser behind it, was served on 31 of 59 sites, challenged on 20, and refused or unanswered on 8. A site that challenges everyone is not enforcing its robots.txt; it is guarding against scripted clients in general.
• **HTTP 200 is not always the full page.** wikihow.com sent ClaudeBot and CCBot a page of 3,038 bytes, against 380,994 bytes for Chrome and GPTBot. We still count it as served, because the answer was a 200 without a challenge, and the bytes are in the CSV.
• **The refusals use different codes.** 403 was the most common, but four sites answered with HTTP 402 (Payment Required) and spiegel.de with a custom 471; reddit.com answered ClaudeBot and CCBot with 429.

The per-request results are public: download the CSV (209 rows) and reuse it under CC BY 4.0 if you credit Crawlerpeek with a link to this page. The 59 sites are a subset of a hand-picked list of large sites, not a random sample of the web.

What the numbers do not show

Our requests only claim to be GPTBot, ClaudeBot or CCBot. They came from one ordinary IP address, not from the operators' crawler ranges, and a CDN may verify a crawler by IP range or reverse DNS. A site that served us the page may refuse the real GPTBot, and a site that refused us may let the real one through. So the finding is narrow and exact: it is what a request that only claims to be the bot receives. That is still what matters for the many scrapers that borrow these names, and for anyone who assumes that a robots.txt line has closed the door. We also tested the home page only, with one request per user agent, and ran no JavaScript. Results change when a site changes its rules.

How to test your own site

Request your home page three ways and compare the status line and the size. Use the user agent strings from our bot pages; these are the ones we used:

# normal browser
curl -sS -o /dev/null -w "%{http_code} %{size_download} %{url_effective}\n" -L -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36" https://example.com/

# GPTBot
curl -sS -o /dev/null -w "%{http_code} %{size_download} %{url_effective}\n" -L -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)" https://example.com/

# ClaudeBot
curl -sS -o /dev/null -w "%{http_code} %{size_download} %{url_effective}\n" -L -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://example.com/

# CCBot
curl -sS -o /dev/null -w "%{http_code} %{size_download} %{url_effective}\n" -L -A "CCBot/2.0 (https://commoncrawl.org/faq/)" https://example.com/

Read the result like this: if your robots.txt disallows the bot and you get 200 with a body about as large as the browser's, the edge is not enforcing the rule. A 403, 402 or 429 means it is. A 200 that is a few kilobytes and contains words like "Just a moment" is a challenge page, which also keeps the bot out but says nothing about your robots.txt. Test /robots.txt the same way: a firewall rule that blocks the bot everywhere also blocks the one file that tells it what you want, and Anthropic warns that blocking its crawlers' IP addresses may not work correctly and can stop them reading your robots.txt.

To see what each bot receives without curl, use see a page as a bot for the page body and the checker on the home page for robots.txt and crawler access in one run. To repeat the check after every change, run it in CI.

Why the two layers disagree

robots.txt and the edge are separate systems with separate owners. Cloudflare's documentation says it plainly: robots.txt compliance is voluntary, the managed file does not prevent crawlers from accessing your content at a technical level, and to enforce access restrictions you use AI Crawl Control to block selected AI crawlers (Cloudflare: robots.txt setting). Typical reasons they drift apart:

• **Someone edited one layer.** The robots.txt came from a CMS plugin or a file in the repository; the CDN rules come from a dashboard nobody connects to it.
• **The edge rule is not about AI crawlers.** A general bot score or rate limit challenges some clients, but a request whose user agent says GPTBot passes it, because the rule never looks at the name.
• **The rule is category-based.** Cloudflare groups bots by purpose (search, agent and training) and can keep a bot that does both, such as a search engine's crawler, out of a training block (Cloudflare: Bot Preference Sync, 21 August 2026). The mixed-use crawlers are excluded from the edge block on purpose.
• **A cache answers first.** A cached copy can be served before a user agent rule is evaluated. Test with a URL you have not requested before.
• **Some bots are not covered by robots.txt at all.** OpenAI says that for ChatGPT-User, which acts on a user's request, robots.txt rules may not apply. A server rule is the only control there (what each operator documents).

How to make them agree

• **Decide the policy per purpose first:** training, search and user-triggered fetching. The guide on allowing AI assistants helps with the trade-offs.
• **Write it into robots.txt**, with the generator if you like, so that crawlers that honor it stop on their own. OpenAI and Anthropic both document that GPTBot and their bots respect it (OpenAI, Anthropic).
• **Enforce the same policy at the edge,** where you want it enforced: in Cloudflare with AI Crawl Control (Cloudflare describes setting allow or block rules for individual crawlers and tracking which ones follow your robots.txt), or with a rule in nginx, Apache or Vercel. On Cloudflare, Bot Preference Sync can generate the robots.txt lines from the same dashboard settings, and the documentation says that Training set to Disallow with the sync on also enables edge blocking for AI training crawlers. Our Cloudflare guide lists the order of steps.
• **Treat the user agent as a claim.** A rule on the name stops honest and lazy clients; a determined scraper changes the name. Check requests against the operator's published IP ranges when it matters (fake crawlers in our logs).
• **Re-run the three curl commands** after each change, and keep them in CI.

Updated 4 November 2026

Data: robots.txt files read on 29 October 2026, requests made on 2 November 2026 between 21:47 and 22:10 UTC. The Cloudflare, OpenAI and Anthropic statements are from their documentation as read on 4 November 2026; vendors change these pages, so check the links.