CCBot (Common Crawl): what it is and how to allow or block it
CCBot is the crawler of Common Crawl, a non-profit that maintains an open repository of web crawl data. Its robots.txt token is CCBot.
Facts
- Operator
- Common Crawl
- Category
- AI training crawler
- robots.txt token
CCBot
- Honors robots.txt?
- Yes, per Common Crawl. Common Crawl documents robots.txt blocking with a User-agent: CCBot rule.
- User-agent example
CCBot/2.0 (https://commoncrawl.org/faq/)
- Official source
- Common Crawl: CCBot
- Published IP ranges
- https://index.commoncrawl.org/ccbot.json
- Compared with the source
What it is used for
Builds an open archive of web crawl data that anyone can use, including AI developers. Collects public pages that may be used to train AI models. Blocking it is a statement about training, not about being found.
What happens if you block it
Blocking it keeps future pages out of new Common Crawl archives. Copies already in earlier archives stay there.
Common Crawl warns that other crawlers falsely identify as CCBot; verify the user agent against the published IP ranges.
How to allow or block CCBot
Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.
User-agent: CCBot Allow: /
User-agent: CCBot Disallow: /
Does your site allow CCBot today? The checker evaluates your robots.txt for this and every other bot.Check my site
What we see on our own sites
No request identifying as CCBot has been recorded in the last 30 days, so there is nothing to show yet. We never fill this in with estimates.
Counts come from the websites we operate; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.
Related bots
Other AI training crawlers
- GPTBot OpenAIOpenAI's crawler for content that may be used to train its generative AI models.
- ClaudeBot AnthropicAnthropic's crawler that collects web content which could contribute to training its generative AI models.
- meta-externalagent MetaMeta's crawler that gathers web content to train AI models or to improve products by indexing.
- Bytespider ByteDanceByteDance's crawler. We found no official ByteDance documentation page for it.
All AI Training Crawlers · Bot directory · Which bots to allow or block · robots.txt generator · See a page as a bot