Anthropic crawlers: what each bot does
Anthropic runs 3 bots that we document: ClaudeBot, Claude-SearchBot, Claude-User. They have different jobs (ai training crawler, ai search crawler, ai user-triggered fetcher), so a robots.txt rule for one does not apply to the others.
| Bot | Category | robots.txt token | Honors robots.txt? | What blocking does |
|---|---|---|---|---|
| ClaudeBot | AI training crawler | ClaudeBot | Yes, per Anthropic | Per Anthropic, blocking it signals that the site's future materials should be excluded from its AI model training datasets. |
| Claude-SearchBot | AI search crawler | Claude-SearchBot | Yes, per Anthropic | Per Anthropic, its system cannot index your content for search optimization, which may reduce your visibility and accuracy in user search results. |
| Claude-User | AI user-triggered fetcher | Claude-User | Token documented by Anthropic | Per Anthropic, disabling it prevents retrieval of your content in response to a user query, which may reduce your visibility in user-directed web search. |
Stay findable, opt out of training
If you want Anthropic's search and fetch bots to keep access but its training bot and control tokens to stay away, this is the robots.txt for it. It is one option, not a recommendation.
User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: ClaudeBot Disallow: /
97 requests from Anthropic bots on our sites in the last 30 days. Bot traffic data