ClaudeBot (Anthropic): what it is and how to allow or block it
ClaudeBot is Anthropic's crawler that collects web content which could contribute to training its generative AI models. Its robots.txt token is ClaudeBot.
Facts
- Operator
- Anthropic
- Category
- AI training crawler
- robots.txt token
ClaudeBot
- Honors robots.txt?
- Yes, per Anthropic. Anthropic documents robots.txt as the opt-out and also supports the non-standard Crawl-delay directive.
- User-agent example
- Contains
claudebot(the operator documents no full example string)
- Official source
- Anthropic: Does Anthropic crawl data from the web?
- Published IP ranges
- https://claude.com/crawling/bots.json
- Compared with the source
What it is used for
Collects public web content that may be used to train Anthropic models. Collects public pages that may be used to train AI models. Blocking it is a statement about training, not about being found.
What happens if you block it
Per Anthropic, blocking it signals that the site's future materials should be excluded from its AI model training datasets.
Anthropic advises against blocking by IP address and recommends robots.txt.
How to allow or block ClaudeBot
Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.
User-agent: ClaudeBot Allow: /
User-agent: ClaudeBot Disallow: /
Does your site allow ClaudeBot today? The checker evaluates your robots.txt for this and every other bot.Check my site
What we see on our own sites
In the last 30 days, 94 requests on 1 day identified as ClaudeBot (latest: ). That is a small sample: read it as an example, not as a measurement of the web.
- Content pages: 72
- robots.txt: 5
- Legal pages: 5
- Images and files: 5
- Home page: 3
- RSS feeds: 2
- Sitemaps: 1
- llms.txt: 1
Counts come from the websites we operate; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.
Related bots
More from Anthropic
- Claude-SearchBot AI search crawlerAnthropic's crawler that analyzes online content to improve the quality of Claude's search results.
- Claude-User AI user-triggered fetcherThe agent Claude uses to open a page when a person asks it a question that needs a website.
Other AI training crawlers
- GPTBot OpenAIOpenAI's crawler for content that may be used to train its generative AI models.
- CCBot Common CrawlThe crawler of Common Crawl, a non-profit that maintains an open repository of web crawl data.
- meta-externalagent MetaMeta's crawler that gathers web content to train AI models or to improve products by indexing.
- Bytespider ByteDanceByteDance's crawler. We found no official ByteDance documentation page for it.
All AI Training Crawlers · Bot directory · Bots from Anthropic · Which bots to allow or block · robots.txt generator · See a page as a bot