Google crawlers: what each bot does
Google runs 2 bots that we document: Google-Extended, Googlebot. They have different jobs (ai control token, search engine crawler), so a robots.txt rule for one does not apply to the others.
| Bot | Category | robots.txt token | Honors robots.txt? | What blocking does |
|---|---|---|---|---|
| Google-Extended | AI control token | Google-Extended | Yes (control token) | Google states that Google-Extended does not impact a site's inclusion in Google Search. |
| Googlebot | Search engine crawler | Googlebot | Yes, per Google | Blocking Googlebot affects Google Search, including its AI features. It is the opposite of what most sites want. |
Stay findable, opt out of training
If you want Google's search and fetch bots to keep access but its training bot and control tokens to stay away, this is the robots.txt for it. It is one option, not a recommendation.
User-agent: Googlebot Allow: / User-agent: Google-Extended Disallow: /
26 requests from Google bots on our sites in the last 30 days. Bot traffic data