Is Googlebot an AI Crawler? What Each Google Token Controls

Googlebot is Google's Search crawler. Google-Extended, a token with no user agent string of its own, covers Gemini training and grounding, not Search.

Published
8 min read
  • googlebot
  • google-extended
  • robots.txt
  • ai overviews
  • gemini
  • ai training opt-out

Googlebot is Google's Search crawler, and Google says AI is built into Search. The Gemini opt-out is a different robots.txt token, Google-Extended, with no user agent string of its own: it appeared 0 times in our full server log from 19 to 31 October 2026, against 629 requests from Googlebot. Blocking the wrong one costs you Search crawling or Gemini use.

As of 1 November 2026.

You added two lines to robots.txt after reading about Gemini, then searched your access log for the name. This invented example shows what you see:

User-agent: Google-Extended
Disallow: /
$ grep -c "Google-Extended" access.log
0
$ grep "Googlebot" access.log | head -1
203.0.113.7 - - [01/Nov/2026:08:14:02 +0000] "GET /pricing HTTP/1.1" 200 5120 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36"

Zero lines for the token you just blocked, and Googlebot all day. Neither fact tells you what is controlled. Google's documents do.

Is Googlebot an AI crawler? What Google's own pages say

Googlebot is the crawler Google uses for Search, and Google documents no separate Googlebot setting for AI features. Google's common crawlers page (checked 1 November 2026) says crawling preferences addressed to Googlebot "affect Google Search (including Discover and all Google Search features), as well as other products such as Google Images, Video mode, Google News, and Discover."

The AI features guide says: "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary." To be a supporting link, "a page must be indexed and eligible to be shown in Google Search with a snippet."

Which use maps to which control? One table, built from Google's own pages, answers that.

Which Google token controls which AI use?

Eight Google uses split across robots.txt tokens, snippet controls and, for Google-Agent, no robots.txt token at all. The table is ours, built on 1 November 2026 from Google's crawler pages, the AI features guide and the robots meta tag documentation.

Google use Crawler Google names Control What Google says it does
Search results Googlebot Googlebot in robots.txt; noindex The Googlebot token affects Search, Images, Video mode, News and Discover. noindex: "Do not show this page, media, or resource in search results."
AI Overviews and AI Mode links None named Googlebot in robots.txt; nosnippet, max-snippet, data-nosnippet, noindex nosnippet "will also prevent the content from being used as a direct input for AI Overviews and AI Mode." max-snippet limits how much may be used.
Gemini model training None (existing user agents) Google-Extended Manages whether content "may be used for training future generations of Gemini models." Does not affect inclusion in Search.
Gemini Apps grounding None (existing user agents) Google-Extended Same token covers "grounding" in Gemini Apps.
Grounding with Google Search on Vertex AI None (existing user agents) Google-Extended Same token and same definition.
Vertex AI Agents built from a site Google-CloudVertexBot Google-CloudVertexBot Affects "crawls requested by the site owners' for building Vertex AI Agents. It has no effect on Google Search or other products."
An agent acting on a user's request Google-Agent No robots.txt token listed User-triggered fetchers "generally ignore robots.txt rules." The entry names the IP file user-triggered-agents.json.
Generic crawling GoogleOther GoogleOther Preferences for it "don't affect any specific product."

Google names no crawler for AI Overviews or AI Mode: its four crawler pages never mention them (read 1 November 2026). The second row is our reading of one Google sentence: "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search." Google never writes that AI Overviews are fetched by Googlebot.

Google-Extended covers training and grounding in one token, and its entry names Gemini Apps and Vertex AI, not AI Overviews or AI Mode.

The table says what Google documents. What do the crawlers do on a real site?

What did Google's crawlers do in our logs from 19 to 31 October 2026?

In our full server log from 19 to 31 October 2026, Googlebot made 629 requests and Google-Extended none. GoogleOther made 1,391, Googlebot-Image 100 and Google-InspectionTool 87, all 1,578 from inside Google's published IP ranges. Of Googlebot's 629, 620 were inside and 9 outside.

Method box.

  • Period: 19 to 31 October 2026, 13 UTC days; our counters start on 19 October 2026.
  • Counted: the full server log, every path including robots.txt and static files (2,713 URL rows, 10,286 requests naming a bot). A verified flag says whether the request IP is inside the operator's published ranges. The last column is our page-request counter, which skips bundled static files (8,236 requests); it carries no verified flag.
  • Limits: one site, 13 days. The two views do not reconcile for every bot. Skipped static files fit Google-InspectionTool (75 of its 87 logged requests are static files; the counter shows 12, matching the 12 that are not). They do not explain GoogleOther: 1,289 of its 1,391 logged requests went to extension-less page paths, against 108 in the counter. Googlebot (639 counter, 629 log) and AdsBot-Google-Mobile (2, none in the log) run the other way. We did not trace these gaps. Logs show who fetched, never what Google later did with the content.
Google token or crawler Full log requests Verified true / false Page-request counter
Googlebot 629 620 / 9 639
GoogleOther 1,391 1,391 / 0 108
Googlebot-Image 100 100 / 0 18
Google-InspectionTool 87 87 / 0 12
AdsBot-Google-Mobile 0 none 2
Google-CloudVertexBot 1 0 / 1 0
Google-Extended 0 none 0
Google-Agent 0 none 0

Google-Extended never appears because it has no user agent string, so the empty grep above says nothing about whether your rule works. Googlebot's 9 unverified requests sit on 5 URL rows, / among them, and a header alone cannot tell you whether those were Google: verifying Googlebot requests shows the checks.

If Google-Extended leaves no trace in a log, what does blocking it do to your search visibility?

Does blocking Google-Extended remove you from AI Overviews?

By Google's documents, no. Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search," and a supporting link in AI Overviews needs a page that is indexed and eligible for Search with a snippet. Google does not join the two in one sentence; that step is our reading.

What you may lose is the Gemini side. Google's blog post of 30 October 2026 says: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." It does not say what applies to a site that names no token, so we claim no effect there.

Staying out of AI Overviews runs through Search controls, and those also change classic results: nosnippet removes the text snippet from classic results too. There is a trap with robots.txt. Google's robots meta documentation says these settings "can be read and followed only if crawlers are allowed to access the pages that include these settings," so a Disallow for Googlebot hides the tag from Google.

Google "intends to support the AIPREF shared machine-readable training preference signal" in early 2027, per its crawler page. The __usage_ai-training token has its own article.

So what does a file that keeps Search and opts out of Gemini look like, and does it parse the way you expect?

How do you keep Search but opt out of Gemini training?

Add one group that names Google-Extended with Disallow: / and leave Googlebot out of the file. In our parser test only Google-Extended is refused and Googlebot stays allowed. We ran Python 3.12.6 urllib.robotparser on 1 November 2026 against the file shown at the top:

Token asked Test A: only Google-Extended disallowed Test B: only Googlebot disallowed
Googlebot True False
Google-Extended False True
GoogleOther True True
Google-CloudVertexBot True True

Each cell is the real can_fetch("https://example.com/article") result. Test A is the file you want. Test B is the mistake: the Googlebot group shuts Search crawling.

Two limits. The standard library parser is not Google's parser: in test B it returns True for Google-CloudVertexBot, although Google's crawler page lists Googlebot as a second token for that crawler. And robots.txt states a preference for compliant crawlers, it enforces nothing. The Googlebot robots.txt guide covers the Search side.

Takeaway: block Google-Extended for Gemini, use snippet controls for AI Overviews, and do not use Disallow on Googlebot as your Gemini opt-out. Next, open the bot catalogue and its Google-Extended entry.

Common follow-up questions

What is the Google-Extended user agent?

None. Google's crawler page says it "doesn't have a separate HTTP request user agent string"; crawling uses existing Google user agent strings. It is a robots.txt token only, so no request in your access log carries its name.

Does Google-Agent obey robots.txt?

Google says user-triggered fetchers "generally ignore robots.txt rules" because a user requested the fetch. The Google-Agent entry lists no robots.txt token, only the IP file user-triggered-agents.json, which is where to check a request's source.

Does blocking Googlebot stop Gemini training?

Not as Google documents it. Google names Google-Extended as the training and grounding control and Googlebot as the Search crawling control, so a Googlebot block cuts Search crawling and is not the documented Gemini opt-out.

Sources

Checked 1 November 2026.