ChatGPT and robots.txt: which OpenAI bot does what, and which obeys your file

Which OpenAI bots does robots.txt control?

OpenAI documents separate user agents for separate jobs. Each has its own token, so a rule for one does not change the others. (Source: OpenAI, Overview of OpenAI crawlers, read 26 October 2026.)

"Each setting is independent of the others," OpenAI writes, and gives this example: allow OAI-SearchBot to appear in search results while disallowing GPTBot so crawled content is not used for training.

What do I put in robots.txt for ChatGPT?

Pick the goal, then copy the matching group. A crawler uses only the most specific group that names it, so each bot needs its own group once you have a User-agent: * group.

Appear in ChatGPT search, opt out of training:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Opt out of both automatic OpenAI crawlers:

User-agent: OAI-SearchBot
Disallow: /

User-agent: GPTBot
Disallow: /

OpenAI recommends allowing OAI-SearchBot in robots.txt, and also allowing requests from its published IP ranges, if you want your site to appear in ChatGPT search. A firewall that blocks those ranges defeats an Allow rule; see step 5 of the allow guide.

Does ChatGPT-User obey robots.txt?

OpenAI does not promise that it does. Its page says ChatGPT-User is used when "users ask ChatGPT or a CustomGPT a question" and it "may visit a web page", that it "is not used for crawling the web in an automatic fashion", and that "because these actions are initiated by a user, robots.txt rules may not apply". It adds that ChatGPT-User "is not used to determine whether content may appear in Search" and tells site owners to use OAI-SearchBot in robots.txt for Search opt-outs and automatic crawl. (Same OpenAI page, read 26 October 2026.)

Read that literally: a Disallow for ChatGPT-User is allowed in the file, but OpenAI does not commit to honouring it, and blocking GPTBot or OAI-SearchBot does not stop the fetcher. Whether it behaves differently on your site is something only your logs can show; the canary-URL test is a way to find out.

What does ChatGPT-User look like on a small site?

We checked the requests that named an OpenAI bot on four small sites on one server between 19 and 25 October 2026 against the IP lists OpenAI publishes. The counts show what a small, new site can expect to see; they say nothing about large sites, and we cannot tell from them whether a robots.txt rule would have been followed.

ChatGPT-User appeared on three of the four sites. 151 of its 167 listed requests (90 percent) asked for a home page; the rest were spread over a handful of other pages, one request each or a few. That fits OpenAI's description: a person asked about a site, and the fetcher opened it. The 38 requests from outside the list borrowed the name; the user agent is only a claim, so do not allow or block on it alone. The 48-hour log study explains the method.

How do I stop ChatGPT-User from reading a page?

Not with robots.txt alone. The options are rules on your own server or CDN. Refuse the user agent ChatGPT-User with a rule such as the nginx one, or the Apache, Cloudflare or Vercel equivalent, and combine it with OpenAI's IP list so a forged name is judged by its address. The cost is that a person who gives ChatGPT your link is told it cannot read the page. A page that must stay private should sit behind a login, which this guide treats as general practice rather than something OpenAI documents.

How do I check that a request is really ChatGPT-User?

Compare the requesting address with https://openai.com/chatgpt-user.json. On 26 October 2026 the file had a creationTime of 2026-10-20 and 234 address prefixes, and OpenAI lists the full user-agent string as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot. Download the list fresh each time because it changes. The IP checker does the comparison for one address, and the user-agent lookup names the bot behind a string from your log.

Checked 26 October 2026

Source: OpenAI's overview of its crawlers. We quote short phrases and paraphrase the rest. OpenAI can change this page; read it again before relying on a detail. Request counts are from our own logs of four small sites on one server, 19 to 25 October 2026, with addresses counted and not stored.