AI crawler user agents: the full list for 2026
This is the list of 51 crawlers we document, grouped by purpose: each with its operator, the token to use in robots.txt, the user-agent string from the operator's documentation and the string we saw in our own logs. A user agent is only a claim, so the last column says whether a request can be checked against the operator's own data.
Compared with the operators' documentation on 2026-10-18. Logged strings are the most common variant on four sites we operate, 19 to 25 Oct 2026; bots we did not see have none.
Download CSVOpen JSONColumns and license (CC BY 4.0)
AI Training Crawlers7
| Bot | Operator | robots.txt token | Documented user agent | User agent we logged | Verifiable |
|---|---|---|---|---|---|
| GPTBot | OpenAI | GPTBot | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot) | YesIP list |
| ClaudeBot | Anthropic | ClaudeBot | Contains claudebot; no full example documented | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) | YesIP list |
| CCBot | Common Crawl | CCBot | CCBot/2.0 (https://commoncrawl.org/faq/) | CCBot/2.0 (https://commoncrawl.org/faq/) | YesIP list |
| meta-externalagent | Meta | meta-externalagent | meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)) | No |
| Bytespider | ByteDance | Bytespider | Contains bytespider; no full example documented | not seen | No |
| Amazonbot | Amazon | Amazonbot | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36 | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36 | YesIP list |
| DeepSeekBot | DeepSeek | DeepSeekBot | Contains deepseekbot; no full example documented | Mozilla/5.0 (compatible; DeepSeekBot/1.0; +https://www.deepseek.com/bot) | No |
AI Search Crawlers6
| Bot | Operator | robots.txt token | Documented user agent | User agent we logged | Verifiable |
|---|---|---|---|---|---|
| OAI-SearchBot | OpenAI | OAI-SearchBot | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot | YesIP list |
| Claude-SearchBot | Anthropic | Claude-SearchBot | Contains claude-searchbot; no full example documented | not seen | YesIP list |
| PerplexityBot | Perplexity | PerplexityBot | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) | Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) | YesIP list |
| ExaSearchBot | Exa | ExaSearchBot | Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/) | Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/) | Yessigned requests |
| ShapBot | Parallel Web Systems | ShapBot | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0 | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0 | YesIP list, signed requests |
| xAI-SearchBot | xAI | xAI-SearchBot | Contains xai-searchbot; no full example documented | Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai) | No |
AI User-Triggered Fetchers6
| Bot | Operator | robots.txt token | Documented user agent | User agent we logged | Verifiable |
|---|---|---|---|---|---|
| ChatGPT-User | OpenAI | ChatGPT-User | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot | YesIP list |
| Claude-User | Anthropic | Claude-User | Contains claude-user; no full example documented | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +claude-user@anthropic.com) | YesIP list |
| Perplexity-User | Perplexity | Perplexity-User | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) | not seen | YesIP list |
| meta-externalfetcher | Meta | meta-externalfetcher | meta-externalfetcher/1.1 | not seen | No |
| DuckAssistBot | DuckDuckGo | DuckAssistBot | DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html) | DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html) | YesIP list |
| Google-CloudVertexBot | Google-CloudVertexBot | Contains google-cloudvertexbot; no full example documented | Mozilla/5.0 (compatible; Google-CloudVertexBot; +https://cloud.google.com/vertex-ai-bot) | YesIP list |
AI Control Tokens2
| Bot | Operator | robots.txt token | Documented user agent | User agent we logged | Verifiable |
|---|---|---|---|---|---|
| Google-Extended | Google-Extended | none (never sent) | n/a | n/a (never sends requests) | |
| Applebot-Extended | Apple | Applebot-Extended | none (never sent) | n/a | n/a (never sends requests) |
Search Engine Crawlers10
| Bot | Operator | robots.txt token | Documented user agent | User agent we logged | Verifiable |
|---|---|---|---|---|---|
| Googlebot | Googlebot | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.52 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | YesIP list, reverse DNS | |
| Bingbot | Microsoft | Bingbot | Contains bingbot; no full example documented | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36 | YesIP list |
| Applebot | Apple | Applebot | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) | YesIP list, reverse DNS |
| Googlebot-Image | Googlebot-Image | Googlebot-Image/1.0 | Googlebot-Image/1.0 | YesIP list | |
| Storebot-Google | Storebot-Google | Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36 | not seen | YesIP list | |
| DuckDuckBot | DuckDuckGo | DuckDuckBot | DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html) | DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html) | YesIP list |
| SofyaBot | Sofya | SofyaBot | SofyaBot/1.0 (+https://sofya.co/bot) | SofyaBot/1.0 (+https://sofya.co/bot) | YesIP list, reverse DNS |
| YandexBot | Yandex | YandexBot | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | Yesreverse DNS |
| Baiduspider | Baidu | Baiduspider | Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) | Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) | No |
| PetalBot | Huawei | PetalBot | Contains petalbot; no full example documented | not seen | No |
SEO Tool Crawlers10
| Bot | Operator | robots.txt token | Documented user agent | User agent we logged | Verifiable |
|---|---|---|---|---|---|
| AhrefsBot | Ahrefs | AhrefsBot | Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/) | Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/) | YesIP list |
| SemrushBot | Semrush | SemrushBot | Contains semrushbot; no full example documented | Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html) | No |
| MJ12bot | Majestic | MJ12bot | Contains mj12bot; no full example documented | Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/) | No |
| serpstatbot | Serpstat | serpstatbot | Contains serpstatbot; no full example documented | serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; abuse@serpstatbot.com) | YesIP list |
| SERankingBacklinksBot | SE Ranking | SERankingBacklinksBot | Mozilla/5.0 (compatible; SERankingBacklinksBot/1.0; +https://seranking.com/backlinks-crawler) | Mozilla/5.0 (compatible; SERankingBacklinksBot/1.0; +https://seranking.com/backlinks-crawler) | Yesreverse DNS |
| QlyzeBot | Qlyze | QlyzeBot | Mozilla/5.0 (compatible; QlyzeBot/1.0; +https://app.qlyze.io/bot) | Mozilla/5.0 (compatible; QlyzeBot/1.0; +https://app.qlyze.io/bot) | No |
| FossickBot | DATAFREAK LTD | FossickBot | Mozilla/5.0 (compatible; FossickBot/1.0; +https://fossick.bot) | Mozilla/5.0 (compatible; FossickBot/1.0; +https://fossick.bot) | Yessigned requests |
| DotBot | Moz | dotbot | Contains dotbot; no full example documented | not seen | No |
| DataForSeoBot | DataForSEO | DataForSeoBot | Mozilla/5.0 (compatible; DataForSeoBot; +https://dataforseo.com/dataforseo-bot) | not seen | YesIP list, reverse DNS |
| SiteCheckerBotCrawler | Sitechecker | SiteCheckerBotCrawler | SiteCheckerBotCrawler/1.0 (+http://sitechecker.pro) | not seen | YesIP list |
Special-Purpose Crawlers5
| Bot | Operator | robots.txt token | Documented user agent | User agent we logged | Verifiable |
|---|---|---|---|---|---|
| GoogleOther | GoogleOther | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36 | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.52 Mobile Safari/537.36 (compatible; GoogleOther) | YesIP list | |
| Google-InspectionTool | Google-InspectionTool | Mozilla/5.0 (compatible; Google-InspectionTool/1.0) | not seen | YesIP list | |
| archive.org_bot | Internet Archive | archive.org_bot | Contains archive.org_bot; no full example documented | Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot) Zeno/7d76126 warc/v0.8.102 | No |
| WebMCPIndexBot | WebMCP Shield | WebMCPIndexBot | WebMCPIndexBot/0.1 (+https://webmcpshield.com/bot) | WebMCPIndexBot/0.1 (+https://webmcpshield.com/bot) | No |
| AgentTrustBot | agenttru.st | AgentTrustBot | Contains agenttrustbot; no full example documented | AgentTrustBot/1.0 (id=95ecd7a1d908; +https://agenttru.st/crawler; purpose=AI agent directory; contact=admin@agenttru.st) | YesIP list, reverse DNS |
Using the list
- To allow or block a bot, use its robots.txt token. The robots.txt generator writes the file.
- To identify a string from your logs, paste it into the user-agent lookup.
- To see whether an address really belongs to a crawler, use the crawler IP checker.
Questions
What is a crawler's user agent?
The User-Agent header is the text a client sends to say what it is. Crawlers put their name in it, usually with a link or email address to a page about the bot. The robots.txt token is a separate, shorter name that you use in a User-agent: line.
Can I trust the user agent in my logs?
No. Any client can send any string, and borrowed bot names are common. Compare the requesting address with the operator's published IP ranges or documented reverse DNS: the crawler IP checker does that, and our study of forged bot names has the numbers.
Which user agents belong to AI crawlers?
The AI groups are training crawlers (such as GPTBot and ClaudeBot), AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User). Control tokens such as Google-Extended appear only in robots.txt, never in logs.
Why does the string I logged differ from the documented one?
Operators change version numbers and browser tokens, and some send several variants. Match on the bot's name inside the string, not on the whole string. The right-hand column shows the most common variant in our own logs for one week.
Social Preview Fetchers5
facebookexternalhitfacebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)SlackbotSlackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)TwitterbotTwitterbot/1.0Twitterbot/1.0TelegramBottelegrambot; no full example documentedTelegramBot (like TwitterBot)LinkedInBotLinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)