OAI-SearchBot (OpenAI): what it is and how to allow or block it
OAI-SearchBot is OpenAI's crawler that surfaces websites in ChatGPT's search features. Its robots.txt token is OAI-SearchBot.
Facts
- Operator
- OpenAI
- Category
- AI search crawler
- robots.txt token
OAI-SearchBot
- Honors robots.txt?
- Yes, per OpenAI. OpenAI documents that robots.txt controls OAI-SearchBot.
- User-agent example
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
- Official source
- OpenAI: Overview of OpenAI crawlers
- Published IP ranges
- https://openai.com/searchbot.json
- Compared with the source
What it is used for
Finds and indexes pages so they can be shown in ChatGPT search answers. Builds the index an AI assistant searches when it answers. Blocking it can keep you out of that assistant's answers.
What happens if you block it
Per OpenAI, sites that opt out will not appear in ChatGPT search answers, though they can still appear as navigational links.
How to allow or block OAI-SearchBot
Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.
User-agent: OAI-SearchBot Allow: /
User-agent: OAI-SearchBot Disallow: /
Does your site allow OAI-SearchBot today? The checker evaluates your robots.txt for this and every other bot.Check my site
What we see on our own sites
In the last 30 days, 5 requests on 1 day identified as OAI-SearchBot (latest: ). That is a small sample: read it as an example, not as a measurement of the web.
- robots.txt: 5
Counts come from the websites we operate; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.
Related bots
More from OpenAI
- GPTBot AI training crawlerOpenAI's crawler for content that may be used to train its generative AI models.
- ChatGPT-User AI user-triggered fetcherThe fetcher ChatGPT uses when a person asks it to open or look at a web page.
Other AI search crawlers
- Claude-SearchBot AnthropicAnthropic's crawler that analyzes online content to improve the quality of Claude's search results.
- PerplexityBot PerplexityPerplexity's crawler that surfaces and links websites in Perplexity search results.
All AI Search Crawlers · Bot directory · Bots from OpenAI · Which bots to allow or block · robots.txt generator · See a page as a bot