Claude-SearchBot (Anthropic): what it is and how to allow or block it
Claude-SearchBot is Anthropic's crawler that analyzes online content to improve the quality of Claude's search results. Its robots.txt token is Claude-SearchBot.
Facts
- Operator
- Anthropic
- Category
- AI search crawler
- robots.txt token
Claude-SearchBot
- Honors robots.txt?
- Yes, per Anthropic. Controlled through robots.txt according to Anthropic.
- User-agent example
- Contains
claude-searchbot(the operator documents no full example string)
- Official source
- Anthropic: Does Anthropic crawl data from the web?
- Published IP ranges
- https://claude.com/crawling/bots.json
- Compared with the source
What it is used for
Navigates the web to improve search result quality by looking at relevance and accuracy of content. Builds the index an AI assistant searches when it answers. Blocking it can keep you out of that assistant's answers.
What happens if you block it
Per Anthropic, its system cannot index your content for search optimization, which may reduce your visibility and accuracy in user search results.
How to allow or block Claude-SearchBot
Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.
User-agent: Claude-SearchBot Allow: /
User-agent: Claude-SearchBot Disallow: /
Does your site allow Claude-SearchBot today? The checker evaluates your robots.txt for this and every other bot.Check my site
What we see on our own sites
No request identifying as Claude-SearchBot has been recorded in the last 30 days, so there is nothing to show yet. We never fill this in with estimates.
Counts come from the websites we operate; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.
Related bots
More from Anthropic
- ClaudeBot AI training crawlerAnthropic's crawler that collects web content which could contribute to training its generative AI models.
- Claude-User AI user-triggered fetcherThe agent Claude uses to open a page when a person asks it a question that needs a website.
Other AI search crawlers
- OAI-SearchBot OpenAIOpenAI's crawler that surfaces websites in ChatGPT's search features.
- PerplexityBot PerplexityPerplexity's crawler that surfaces and links websites in Perplexity search results.
All AI Search Crawlers · Bot directory · Bots from Anthropic · Which bots to allow or block · robots.txt generator · See a page as a bot