GPTBot (OpenAI): what it is and how to allow or block it
GPTBot is OpenAI's crawler for content that may be used to train its generative AI models. Its robots.txt token is GPTBot.
Facts
- Operator
- OpenAI
- Category
- AI training crawler
- robots.txt token
GPTBot
- Honors robots.txt?
- Yes, per OpenAI. OpenAI documents that robots.txt controls GPTBot.
- User-agent example
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
- Official source
- OpenAI: Overview of OpenAI crawlers
- Published IP ranges
- https://openai.com/gptbot.json
- Compared with the source
What it is used for
Crawls public pages that may be used to train OpenAI's foundation models. Collects public pages that may be used to train AI models. Blocking it is a statement about training, not about being found.
What happens if you block it
Signals that your content should not be used for that training. It does not decide whether you appear in ChatGPT search; that is OAI-SearchBot.
How to allow or block GPTBot
Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.
User-agent: GPTBot Allow: /
User-agent: GPTBot Disallow: /
Does your site allow GPTBot today? The checker evaluates your robots.txt for this and every other bot.Check my site
What we see on our own sites
In the last 30 days, 158 requests on 1 day identified as GPTBot (latest: ).
- Content pages: 134
- Legal pages: 11
- Images and files: 5
- Home page: 4
- RSS feeds: 2
- Sitemaps: 1
- llms.txt: 1
Counts come from the websites we operate; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.
Related bots
More from OpenAI
- OAI-SearchBot AI search crawlerOpenAI's crawler that surfaces websites in ChatGPT's search features.
- ChatGPT-User AI user-triggered fetcherThe fetcher ChatGPT uses when a person asks it to open or look at a web page.
Other AI training crawlers
- ClaudeBot AnthropicAnthropic's crawler that collects web content which could contribute to training its generative AI models.
- CCBot Common CrawlThe crawler of Common Crawl, a non-profit that maintains an open repository of web crawl data.
- meta-externalagent MetaMeta's crawler that gathers web content to train AI models or to improve products by indexing.
- Bytespider ByteDanceByteDance's crawler. We found no official ByteDance documentation page for it.
All AI Training Crawlers · Bot directory · Bots from OpenAI · Which bots to allow or block · robots.txt generator · See a page as a bot