AI user-triggered fetchers: bots that open a page on request
AI user-triggered fetchers retrieve a single page because a person asked an assistant about it, or because an answer needs it. Several operators say robots.txt may not apply to them.
Opens one page because a person (or an answer being generated) needs it. The operator may treat robots.txt differently for these requests.
5 bots in this group
| Bot | Owner | robots.txt token | Honors robots.txt? | Requests on our sites (30 days) |
|---|---|---|---|---|
| ChatGPT-User | OpenAI | ChatGPT-User | May not apply (user-initiated) | none yet |
| Claude-User | Anthropic | Claude-User | Token documented by Anthropic | 3 |
| Perplexity-User | Perplexity | Perplexity-User | Generally ignored (user-initiated) | none yet |
| meta-externalfetcher | Meta | meta-externalfetcher | May bypass (user-initiated) | none yet |
| DuckAssistBot | DuckDuckGo | DuckAssistBot | Yes, per DuckDuckGo | none yet |
3 requests in this group on our sites in the last 30 days.
Block the whole group
Paste into robots.txt to ask every bot in this group to stay away. Only do this if you mean it: read what blocking changes first.
User-agent: ChatGPT-User Disallow: / User-agent: Claude-User Disallow: / User-agent: Perplexity-User Disallow: / User-agent: meta-externalfetcher Disallow: / User-agent: DuckAssistBot Disallow: /