Crawlerpeek
Check a site
← Crawlerpeek

Which AI Bots Hit a New Site in Week One?

On one new site, OAI-SearchBot made 61% of 2,839 bot requests in seven days, ahead of Googlebot and ClaudeBot. Full counts and what each bot is for.

Redaktion Crawlerpeek
4 min read
  • ai-crawlers
  • oai-searchbot
  • server-logs

OpenAI's OAI-SearchBot was by far the most active bot on a brand-new site in its first week: 1,744 of 2,839 bot requests (61%) between 19 and 25 October 2026. Googlebot came second with 355 requests, followed by ClaudeBot with 168. We counted request volume, not arrival order, so we cannot say which bot visited first. This is one site and one week, so treat it as a field report, not a benchmark.

What we measured

We count requests by user-agent on crawlerpeek.com itself, using the bot log behind our AI bot data page. The window is 19 to 25 October 2026 (seven days; 25 October is a partial day). The sample is a single site. We classify bots by the user-agent string a request announces. We did not verify IP ranges for this count, so a share of any bot's hits could be impersonators.

Bot Requests Share of 2,839 What its operator says it is for
OAI-SearchBot 1,744 61.4% Surfacing sites in ChatGPT search results
Googlebot 355 12.5% Google Search crawling
ClaudeBot 168 5.9% Collecting web content that may contribute to model training
AhrefsBot 136 4.8% Powering the Ahrefs and Yep databases
GPTBot 103 3.6% Crawling content that may be used to train OpenAI models
Amazonbot 94 3.3% Improving Amazon products and services, may train Amazon AI models
MJ12bot 66 2.3% Distributed web crawl for a search index and Majestic data
ChatGPT-User 60 2.1% Fetching pages when a ChatGPT user asks
All other bots 113 4.0% not broken out

Counts are exact for the window; shares are rounded. The eight named bots add up to 2,726 requests.

Why OAI-SearchBot came first

We can describe what happened, not why. OpenAI documents OAI-SearchBot as the crawler that surfaces websites in ChatGPT's search features, and says that sites opted out of it will not be shown in ChatGPT search answers. A new site that allows it is a candidate for that index right away. OpenAI's documentation also says that if a site allows both OAI-SearchBot and GPTBot, it may use the results of one crawl for both purposes. Whether that explains the gap between 1,744 and 103 requests on this site is not something the logs can tell us.

One practical point from the same page: robots.txt changes can take about 24 hours to be reflected in OpenAI's search systems. See the OpenAI operator page for its full bot list.

Search, training and user-triggered bots behave differently

The table mixes three jobs, and that matters for what you decide:

  • Search and index crawlers (OAI-SearchBot, Googlebot) decide whether you appear in answers or results. Blocking them removes you from that product.
  • Training crawlers (ClaudeBot, GPTBot) collect content for models. Anthropic says that restricting ClaudeBot signals that a site's future materials should be excluded from its training datasets. OpenAI says the same for GPTBot.
  • User-triggered fetchers (ChatGPT-User) visit a page because a person asked. OpenAI notes that ChatGPT-User is not used to crawl the web automatically and that robots.txt rules may not apply to it.

Each OpenAI setting is independent: you can allow OAI-SearchBot and disallow GPTBot. Anthropic likewise lists three separate robots (ClaudeBot, Claude-User, Claude-SearchBot). Our ClaudeBot page and Googlebot page show the user-agent strings and verification methods we track.

What the SEO-tool bots add

AhrefsBot (136) and MJ12bot (66) are commercial index crawlers, not AI assistants. Ahrefs states that AhrefsBot obeys robots.txt and crawl delay. Majestic states that MJ12bot adheres to robots.txt and that its crawlers are distributed, so it cannot publish a fixed IP range. Together they made 202 requests, more than GPTBot and ChatGPT-User combined (163).

How to run this count on your own logs

Count requests per user-agent token for a seven-day window. For an nginx combined log:

grep -E "OAI-SearchBot|Googlebot|ClaudeBot|AhrefsBot|GPTBot|Amazonbot|MJ12bot|ChatGPT-User" access.log \
  | grep -oE "OAI-SearchBot|Googlebot|ClaudeBot|AhrefsBot|GPTBot|Amazonbot|MJ12bot|ChatGPT-User" \
  | sort | uniq -c | sort -rn

Then compare with robots.txt. If you want AI search visibility but no training use, our guide to allowing AI assistants shows the rules per bot. To confirm that a hit is real, check the operator's published IP list where one exists; user-agent strings alone prove nothing.

Common follow-up questions

Which AI bot crawls a new site first? On this site OAI-SearchBot made the most requests (61%) in seven days; we did not record which bot arrived first. One site is not a rule; your first visitor depends on what links to you and what you allow.

Is GPTBot the same as OAI-SearchBot? No. OpenAI documents them as separate robots with separate robots.txt tokens: OAI-SearchBot for search, GPTBot for training content.

Does ChatGPT-User obey robots.txt? OpenAI says robots.txt rules may not apply, because the request is initiated by a user. Use OAI-SearchBot rules to manage search and automatic crawling.

Why does Googlebot show fewer requests than OAI-SearchBot? We do not know. We only counted requests; the logs do not show crawl intent. Googlebot discovers URLs mainly from links in pages it already crawled, according to Google's documentation, which is one factor on a site with few inbound links.

Do these numbers include fake bots? Possibly. We counted announced user-agents without verifying IP ranges.

Sources

Last checked: 25 October 2026