Crawlerpeek
Check a site

How to block AI bots in nginx by user agent

In nginx, match the User-Agent header with a `map` block and return 403 for the bots you want to refuse. This stops bots that ignore robots.txt, but it only works against honest user agents: a bot can send any string.

By the Crawlerpeek editorial team · Updated · How we check

The rule

Put the map in the http block and the check in the server block. Adjust the list to the tokens you actually want to refuse (each bot page shows its token).

# http { ... }
map $http_user_agent $block_ai_bot {
    default 0;
    ~*(GPTBot|ClaudeBot|CCBot|Bytespider|Amazonbot|meta-externalagent) 1;
}

# server { ... }
if ($block_ai_bot) {
    return 403;
}

Reload with nginx -t && nginx -s reload, then test: curl -I -A "GPTBot" https://example.com/ should return 403 and the same request without the header should return 200.

Do robots.txt first

Operators document robots.txt as the opt-out for their crawlers, and a server block can look like an error to them. Use a server rule for bots that ignore robots.txt or that you cannot reach otherwise. Never block search crawlers you depend on, and note that blocking a search bot such as OAI-SearchBot removes you from that assistant's answers.

Limits

  • A user agent is only a claim, so this stops honest bots and nothing else. Operators publish IP lists for verification where they offer one; see the bot pages.
  • The tokens in a regular expression match anywhere in the header, so check them against your real traffic before you deploy.
  • Blocking by IP is discouraged by some operators (Anthropic recommends robots.txt, Semrush says its IPs are not consecutive ranges).

Reference: nginx documents the `map` directive and the $http_ variables. Other servers: Apache, Cloudflare, Vercel.

See how your own site does.Run the checker

Keep reading