AI search crawlers: the bots behind AI answers
AI search crawlers build the index that an AI assistant searches when it answers a question. If one is blocked, the assistant may not show your pages.
Builds the index an AI assistant searches when it answers. Blocking it can keep you out of that assistant's answers.
3 bots in this group
| Bot | Owner | robots.txt token | Honors robots.txt? | Requests on our sites (30 days) |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | OAI-SearchBot | Yes, per OpenAI | 5 |
| Claude-SearchBot | Anthropic | Claude-SearchBot | Yes, per Anthropic | none yet |
| PerplexityBot | Perplexity | PerplexityBot | Yes, per Perplexity | none yet |
5 requests in this group on our sites in the last 30 days.
Block the whole group
Paste into robots.txt to ask every bot in this group to stay away. Only do this if you mean it: read what blocking changes first.
User-agent: OAI-SearchBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: PerplexityBot Disallow: /