How to block AI crawlers in robots.txt: tokens for training, search and user requests

Which AI crawler tokens do I need to block?

Each operator has its own tokens, and a rule for one does not change another. The table lists what each operator documents. Sources: the operators' own pages, read on 26 October 2026 and linked in the last section.

Two tokens never appear in your logs. Google-Extended and Applebot-Extended have no user agent of their own; the crawling is done by Googlebot and Applebot, and the token only states what the content may be used for. Bytespider (ByteDance) is widely listed as a training crawler, but we found no official page about its robots.txt behaviour, so it is not in the snippets below. Add it only if you see it in your logs and accept that its effect is unconfirmed.

What is the robots.txt to block all AI crawlers?

This blocks the documented training tokens and the AI search crawlers. Repeating User-agent lines lets several tokens share one set of rules, as the robots.txt specification allows.

# Training and data collection
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
Disallow: /

# AI search crawlers (you will not be shown in these answers)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /

Put the groups in your robots.txt at the root of every host you want covered; blog.example.com needs its own file. Keep any existing User-agent: * group: a crawler that has its own group ignores the * group, so the groups above do not change the rules for other bots. To build and download a file with these tokens, use the robots.txt generator.

Can I block training but keep AI search?

Yes, and that is the opposite decision from the one above. Delete the second group and keep the first, which is exactly what how to let AI assistants read your site walks through, including why the search crawlers should be allowed and how to check that your firewall does not block them.

How do I block user-triggered AI fetchers?

These bots open a page because a person asked an assistant about it. Operators treat them differently, so robots.txt is an unreliable control here.

User-agent: ChatGPT-User
User-agent: Perplexity-User
User-agent: Claude-User
User-agent: meta-externalfetcher
Disallow: /

Add this group if you want to state the intent. For the three fetchers whose operators say robots.txt may not apply, enforce it on the server (next section). The cost is real: a person who gives an assistant your link is told it cannot read the page. The canary-URL test shows what a fetcher does on your own site.

What does each block cost?

Do not block Googlebot to keep AI out of Google. Google says robots.txt rules for Googlebot are the control for how your site is crawled for Search, where AI Overviews and AI Mode are built in, and points to nosnippet, data-nosnippet, max-snippet and noindex to limit what is shown. Blocking Googlebot removes you from Search crawling altogether; see Googlebot and robots.txt.

What if a crawler ignores robots.txt?

robots.txt is a request, not a lock. Where an operator documents compliance, a rule works. Where it does not, or where a fetcher is user-triggered, refuse the request on your server or CDN. Pick the guide for your stack:

• nginx user-agent rules
• Apache .htaccess rules
• Cloudflare
• Vercel AI bots ruleset

A user agent is only a claim, so judge a request by its IP address against the operator's published list where one exists (fake crawlers and how to check them). Anthropic advises against blocking its IP addresses, because that stops the bots from reading robots.txt and may not reliably guarantee an opt-out.

How do I check that the block works?

• Fetch https://yourdomain/robots.txt and confirm a 200 status and plain text (how to test your file).
• Run your address through the checker. It evaluates your robots.txt for every bot in the table above and shows which ones may fetch the page.
• Allow time: OpenAI and Perplexity each say a robots.txt change can take about 24 hours to be picked up.
• Look in your access log for the tokens afterwards. Requests that name a blocked bot from an address outside the operator's list are forged, not the real crawler.

Sources, read 26 October 2026

OpenAI, overview of its crawlers; Anthropic, does Anthropic crawl data from the web; Perplexity, bots and user agents; Google, common crawlers (updated 15 October 2026) and AI features and your website (updated 10 December 2025); Apple, About Applebot; Meta, web crawlers; Amazon, Amazonbot; Common Crawl, CCBot. Several pages show no date (marked undated where we quote them: Amazon, Meta, OpenAI). We paraphrase and quote short phrases. Operators add and rename bots, so re-read the page of any bot you rely on.