Claude-User (Anthropic): what it is and how to allow or block it
Claude-User is the agent Claude uses to open a page when a person asks it a question that needs a website. Its robots.txt token is Claude-User.
Facts
- Operator
- Anthropic
- Category
- AI user-triggered fetcher
- robots.txt token
Claude-User
- Honors robots.txt?
- Token documented by Anthropic. Anthropic lists a robots.txt token for it.
- User-agent example
- Contains
claude-user(the operator documents no full example string)
- Official source
- Anthropic: Does Anthropic crawl data from the web?
- Published IP ranges
- https://claude.com/crawling/bots.json
- Compared with the source
What it is used for
Supports Claude users: when someone asks Claude a question, it may access websites with this agent. Opens one page because a person (or an answer being generated) needs it. The operator may treat robots.txt differently for these requests.
What happens if you block it
Per Anthropic, disabling it prevents retrieval of your content in response to a user query, which may reduce your visibility in user-directed web search.
How to allow or block Claude-User
Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.
User-agent: Claude-User Allow: /
User-agent: Claude-User Disallow: /
Does your site allow Claude-User today? The checker evaluates your robots.txt for this and every other bot.Check my site
What we see on our own sites
In the last 30 days, 3 requests on 1 day identified as Claude-User (latest: ). That is a small sample: read it as an example, not as a measurement of the web.
- Home page: 1
- robots.txt: 1
- Legal pages: 1
Counts come from the websites we operate; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.
Related bots
More from Anthropic
- ClaudeBot AI training crawlerAnthropic's crawler that collects web content which could contribute to training its generative AI models.
- Claude-SearchBot AI search crawlerAnthropic's crawler that analyzes online content to improve the quality of Claude's search results.
Other AI user-triggered fetchers
- ChatGPT-User OpenAIThe fetcher ChatGPT uses when a person asks it to open or look at a web page.
- Perplexity-User PerplexityThe fetcher Perplexity uses to visit a page when a person's question needs it.
- meta-externalfetcher MetaMeta's fetcher that retrieves individual URLs when a user asks, for example to help an AI complete a task.
- DuckAssistBot DuckDuckGoDuckDuckGo's crawler that fetches pages in real time for its AI-assisted answers.
All AI User-Triggered Fetchers · Bot directory · Bots from Anthropic · Which bots to allow or block · robots.txt generator · See a page as a bot