AI training crawlers: which bots collect pages for model training
AI training crawlers are bots that collect public web pages which AI companies may use to train their models. Blocking them is a decision about training; it does not remove you from AI search.
Collects public pages that may be used to train AI models. Blocking it is a statement about training, not about being found.
6 bots in this group
| Bot | Owner | robots.txt token | Honors robots.txt? | Requests on our sites (30 days) |
|---|---|---|---|---|
| GPTBot | OpenAI | GPTBot | Yes, per OpenAI | 158 |
| ClaudeBot | Anthropic | ClaudeBot | Yes, per Anthropic | 94 |
| CCBot | Common Crawl | CCBot | Yes, per Common Crawl | none yet |
| meta-externalagent | Meta | meta-externalagent | Yes, per Meta | none yet |
| Bytespider | ByteDance | Bytespider | Reports differ | none yet |
| Amazonbot | Amazon | Amazonbot | Yes, per Amazon | none yet |
252 requests in this group on our sites in the last 30 days.
Block the whole group
Paste into robots.txt to ask every bot in this group to stay away. Only do this if you mean it: read what blocking changes first.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: meta-externalagent Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Disallow: /