Do ChatGPT, Claude and Gemini obey your robots.txt? What operators document, and a test for your logs

Do AI assistants obey robots.txt? It depends on the fetcher

robots.txt was designed for crawlers that fetch pages automatically. When a person pastes a link into an assistant and asks about it, the operators treat the fetch as that person's request, closer to a browser than to a crawler. Google says this directly: because a user requested the fetch, its user-triggered fetchers “generally ignore robots.txt rules”. Perplexity gives the same reason for Perplexity-User and OpenAI says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. So a Disallow rule that works on a search or training crawler can have no effect on the fetcher behind a single question.

The four kinds of AI fetcher

• Training crawlers, such as GPTBot and ClaudeBot, collect pages that may be used to train models. Their operators document robots.txt as the opt-out.
• Search crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, build the index an assistant searches. Blocking them can remove you from that assistant's answers.
• User-triggered fetchers, such as ChatGPT-User, Claude-User and Perplexity-User, open one page because a person asked. This is the group where operators say robots.txt may not apply. See the category page.
• Agents act on a person's behalf. Google's Google-Agent is used by agents hosted on Google infrastructure to browse and act on user request, and Google lists it among its user-triggered fetchers.

The first three kinds are also explained in how to let AI assistants read your site.

What each operator documents about robots.txt

Three details matter when you read this table. First, Anthropic's article speaks of its bots as a group and does not describe Claude-User separately, so the safe reading is “Anthropic states that its bots honor robots.txt”, and your logs can confirm it. Second, Google documents Gemini-related fetchers (Google-GeminiNotebook, Google-Agent) as user-triggered. We did not find a Google page that says which fetcher opens a link pasted into a Gemini chat, so we make no claim about that case. Googlebot itself follows robots.txt, and the Google-Extended token controls whether content Google crawled may be used for Gemini training and grounding. Third, “generally” and “may not” are the operators' own hedges. They describe a policy that can change, so treat the table as dated (see the end of this page).

How to test it on your own site with a canary URL

A canary is a page nobody knows about except you. If a fetcher requests it, the only way it learned the address is that you told an assistant. Two pages are enough: one blocked in robots.txt and one allowed as a control.

• Create two pages with random, unlinked addresses that are not in your sitemap, for example /canary-blocked-x4v8d1 and /canary-allowed-k7q2m9. Put a different made-up sentence on each, so you can tell later whether an answer really came from the page.
• Block only the first one in robots.txt. Repeat the rule in a group for each user-triggered fetcher you want to test, because a bot uses only its most specific group and ignores * once it has its own.
• Wait at least 24 hours. OpenAI and Perplexity both say a robots.txt change can take up to about a day to take effect, and Google caches the file for up to 24 hours.
• In a new chat with web access switched on, ask each assistant to open each address, for example “Open https://example.com/canary-blocked-x4v8d1 and tell me what it says.” Do it once per assistant, and note the time.
• Read your access log (next section).

User-agent: *
Disallow: /canary-blocked-x4v8d1

User-agent: ChatGPT-User
Disallow: /canary-blocked-x4v8d1

User-agent: Claude-User
Disallow: /canary-blocked-x4v8d1

User-agent: Perplexity-User
Disallow: /canary-blocked-x4v8d1

Which log lines to look at

These commands assume nginx or Apache in the default combined format. On Apache, replace the path with /var/log/apache2/access.log* (Debian and Ubuntu) or your own. zgrep reads plain and compressed rotated logs. Behind a CDN or proxy, the first field is the proxy's address unless you log the client IP, so fix that first or the IP check below is meaningless. The first command lists every canary request together with every robots.txt request, in log order:

# every canary request and every robots.txt request, in log order: time, IP, request, status, user agent
zgrep -hE 'canary-|GET /robots\.txt' /var/log/nginx/access.log* \
  | awk -F'"' '{split($1, a, " "); print a[4], a[1], $2, substr($3, 2, 3), $6}'

The second counts which user agents asked for the blocked page:

# which user agents asked for the blocked canary, and how often
zgrep -h 'canary-blocked-x4v8d1' /var/log/nginx/access.log* | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

How to read the result

One run proves little. Operators change behavior, assistants pick different tools for different prompts, and robots.txt is cached. Repeat the test a day later, write down the date and the user-agent version string, and compare the same fetcher over time rather than assistants against each other. For Claude specifically, see what Anthropic's bots and Brave Search each contribute.

Verify the IP address against the operator's list

OpenAI, Anthropic, Perplexity and Google publish their address ranges as JSON files. Every file we checked has a prefixes list of ipv4Prefix and ipv6Prefix entries. This script checks one address against one list; the longer script in the 48-hour log guide bundles several lists. Run it as python3 in_list.py https://openai.com/chatgpt-user.json 203.0.113.7 (that address is a documentation example).

# in_list.py: python3 in_list.py <list-url> <ip>
import ipaddress, json, sys, urllib.request

url, ip = sys.argv[1], ipaddress.ip_address(sys.argv[2])
req = urllib.request.Request(url, headers={"User-Agent": "in-list-check"})
prefixes = json.load(urllib.request.urlopen(req))["prefixes"]
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in prefixes]
print("in list" if any(ip in n for n in nets if n.version == ip.version) else "NOT in list")

The list URLs are in the table above, with https:// in front. Anthropic publishes one list for its bots, so it cannot tell Claude-User from ClaudeBot. Google's user-triggered fetchers have their own files, and Google-Agent uses user-triggered-agents.json. For Google you can also use the reverse DNS check Google describes. All operators' lists change, so download them fresh each time. The machine-readable bot list on this site shows each bot's token and the page where its operator documents it.

If you need to block them anyway

For a fetcher the operator says ignores robots.txt, the rule has to live on your server or at your CDN. Refuse the user agent with a rule such as the one in nginx or Apache, or use Cloudflare or Vercel, and where an operator publishes IP ranges, combine the user agent with the range check so that forged names do not slip through or get blocked by mistake. The cost is that a person who gives the assistant your link will be told it cannot read the page, so use this for pages that must not be fetched, not as a general policy. If you want the opposite, see how to let AI assistants read your site.

Checked 25 October 2026

Sources: OpenAI's overview of its crawlers, Anthropic's article on whether it crawls the web, Perplexity's bots and user agents and Google's pages on user-triggered fetchers and Google-Agent. We paraphrase them and quote only short phrases. These pages change, so check the original before you rely on a detail. We have not run the canary test against these assistants for this page, so it contains no test results of our own.