OpenAI crawlers: what each bot does
OpenAI runs 3 bots that we document: GPTBot, OAI-SearchBot, ChatGPT-User. They have different jobs (ai training crawler, ai search crawler, ai user-triggered fetcher), so a robots.txt rule for one does not apply to the others.
| Bot | Category | robots.txt token | Honors robots.txt? | What blocking does |
|---|---|---|---|---|
| GPTBot | AI training crawler | GPTBot | Yes, per OpenAI | Signals that your content should not be used for that training. It does not decide whether you appear in ChatGPT search; that is OAI-SearchBot. |
| OAI-SearchBot | AI search crawler | OAI-SearchBot | Yes, per OpenAI | Per OpenAI, sites that opt out will not appear in ChatGPT search answers, though they can still appear as navigational links. |
| ChatGPT-User | AI user-triggered fetcher | ChatGPT-User | May not apply (user-initiated) | OpenAI says it is not used for automatic crawling and does not decide whether you appear in search. |
Stay findable, opt out of training
If you want OpenAI's search and fetch bots to keep access but its training bot and control tokens to stay away, this is the robots.txt for it. It is one option, not a recommendation.
User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: GPTBot Disallow: /
163 requests from OpenAI bots on our sites in the last 30 days. Bot traffic data