Web crawlers and bots: user agents and robots.txt tokens
A web crawler is an automated program that fetches web pages for a company or tool. This directory covers 24 bots in 7 groups: AI training, AI search and user-triggered bots, search engines, SEO tools and social previews. Each entry states the robots.txt token, the owner, what blocking does and the official source, compared on 2026-10-18.
Browse by owner: Anthropic, Meta, OpenAI, Apple, Google, Perplexity. Or check your own site.
Use the list in your own work: CSV, JSON or column guide and citation (CC BY 4.0).
AI Training Crawlers6
AI training crawlers are bots that collect public web pages which AI companies may use to train their models. Blocking them is a decision about training; it does not remove you from AI search.
| Bot | Owner | robots.txt token | Honors robots.txt? | Requests on our sites (30 days) |
|---|---|---|---|---|
| GPTBot | OpenAI | GPTBot | Yes, per OpenAI | 158 |
| ClaudeBot | Anthropic | ClaudeBot | Yes, per Anthropic | 70 |
| CCBot | Common Crawl | CCBot | Yes, per Common Crawl | none yet |
| meta-externalagent | Meta | meta-externalagent | Yes, per Meta | none yet |
| Bytespider | ByteDance | Bytespider | Reports differ | none yet |
| Amazonbot | Amazon | Amazonbot | Yes, per Amazon | none yet |
AI Search Crawlers3
AI search crawlers build the index that an AI assistant searches when it answers a question. If one is blocked, the assistant may not show your pages.
| Bot | Owner | robots.txt token | Honors robots.txt? | Requests on our sites (30 days) |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | OAI-SearchBot | Yes, per OpenAI | 5 |
| Claude-SearchBot | Anthropic | Claude-SearchBot | Yes, per Anthropic | none yet |
| PerplexityBot | Perplexity | PerplexityBot | Yes, per Perplexity | none yet |
AI User-Triggered Fetchers5
AI user-triggered fetchers retrieve a single page because a person asked an assistant about it, or because an answer needs it. Several operators say robots.txt may not apply to them.
| Bot | Owner | robots.txt token | Honors robots.txt? | Requests on our sites (30 days) |
|---|---|---|---|---|
| ChatGPT-User | OpenAI | ChatGPT-User | May not apply (user-initiated) | none yet |
| Claude-User | Anthropic | Claude-User | Token documented by Anthropic | 3 |
| Perplexity-User | Perplexity | Perplexity-User | Generally ignored (user-initiated) | none yet |
| meta-externalfetcher | Meta | meta-externalfetcher | May bypass (user-initiated) | none yet |
| DuckAssistBot | DuckDuckGo | DuckAssistBot | Yes, per DuckDuckGo | none yet |
AI Control Tokens2
AI control tokens are names you can use in robots.txt to tell a company how crawled content may be used. They are not separate crawlers, so they never show up in your server logs.
| Bot | Owner | robots.txt token | Honors robots.txt? | Requests on our sites (30 days) |
|---|---|---|---|---|
| Google-Extended | Google-Extended | Yes (control token) | n/a (not in logs) | |
| Applebot-Extended | Apple | Applebot-Extended | Yes (control token) | n/a (not in logs) |
Search Engine Crawlers3
Search engine crawlers fetch pages for a search index. Blocking them removes you from that search engine and from the AI features built on its index.
SEO Tool Crawlers3
SEO tool crawlers collect link and site data for marketing platforms. They do not influence Google rankings, and blocking them only removes your site from those tools' databases.
| Bot | Owner | robots.txt token | Honors robots.txt? | Requests on our sites (30 days) |
|---|---|---|---|---|
| AhrefsBot | Ahrefs | AhrefsBot | Yes, per Ahrefs | none yet |
| SemrushBot | Semrush | SemrushBot | Yes, per Semrush | none yet |
| MJ12bot | Majestic | MJ12bot | Yes, per Majestic | none yet |
Social Preview Fetchers
Social preview fetchers load a page when someone shares its link and build the title, description and image card. Blocking them breaks link previews.
| Bot | Owner | robots.txt token | Honors robots.txt? | Requests on our sites (30 days) |
|---|---|---|---|---|
| facebookexternalhit | Meta | facebookexternalhit | May bypass (integrity checks) | none yet |
| Slackbot | Slack | Slackbot | No, per Slack | none yet |
Cite this list
If you use the list in an article, a README or a tool, please credit it and link to crawlerpeek.com/data/ai-bots.
Crawlerpeek. (2026). AI bot reference list: operators, purposes, user agents and robots.txt tokens (Version 2026-10-18) [Data set]. https://crawlerpeek.com/data/ai-bots
Data: <a href="https://crawlerpeek.com/data/ai-bots">Crawlerpeek AI bot reference list</a> (CC BY 4.0)
License: our compilation and descriptions are CC BY 4.0. Bot names, user-agent tokens and the linked official pages belong to their operators and are not covered by this license.
Not sure which to allow? Read how to let AI assistants read your site or build a file with the robots.txt generator.