Crawlerpeek
Check a site

How to block AI bots in Apache with .htaccess

In Apache, use mod_rewrite to match the User-Agent header and answer named bots with 403 Forbidden. Like all user-agent blocking it only stops bots that send their real name, so keep the matching robots.txt rule too.

By the Crawlerpeek editorial team · Updated · How we check

The rule

Add this to the .htaccess file in your document root (with mod_rewrite enabled and AllowOverride permitting it) or to the virtual host. Use the tokens from the bot directory.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|Amazonbot|meta-externalagent) [NC]
RewriteRule .* - [F,L]

[NC] makes the match case-insensitive and [F] answers 403. Test with curl -I -A "GPTBot" https://example.com/; it should return 403, while a plain curl -I returns 200.

Limits and good practice

  • Do not block the search bots you want to be found by; see which bots to allow.
  • A forged user agent passes straight through. Where an operator publishes IP ranges you can verify against them.
  • Keep a robots.txt rule for the same bots: it is the documented opt-out.

Reference: Apache mod_rewrite documentation. Other platforms: nginx, Cloudflare, Vercel.

See how your own site does.Run the checker

Keep reading