How to block AI crawlers on Cloudflare
Cloudflare's AI Crawl Control lets you allow or block individual AI crawlers and track which ones follow your robots.txt. Cloudflare says it is available on all plans. Use it together with robots.txt, and check that it does not block the search bots you want.
By the Crawlerpeek editorial team · Updated · How we check
What the feature does
According to Cloudflare's documentation, AI Crawl Control (formerly AI Audit) lets you see which AI services access your content and manage that access: you can monitor crawler activity, set allow or block rules for individual crawlers, and track which crawlers follow your robots.txt directives. A pay-per-crawl option is listed as a private beta.
How Cloudflare classifies bots
Cloudflare says it lets all customers manage three AI-related use cases: search, agent and training, and that a single bot can show more than one behavior (Cloudflare bots concepts). This maps closely to the categories used here: training crawlers, search crawlers and user-triggered fetchers.
Recommended order
- Decide per category: usually allow AI search and user-triggered bots, and choose separately about training (see the guide).
- Set the same intent in robots.txt: the generator writes it for you.
- Review the crawler list in the Cloudflare dashboard so that a block rule does not catch a search bot you rely on.
- Afterwards run the checker from outside to confirm that bots you allowed in robots.txt are not refused at the edge.
Cloudflare's documentation does not give a menu path, so look for the feature names above in your dashboard.
See how your own site does.Run the checker