Googlebot and robots.txt: the token, the rules, and how to verify a request

Does Googlebot obey robots.txt?

Yes. Google's list of common crawlers, which includes Googlebot, says they "always obey robots.txt rules when crawling automatically". Google reads the file at the top of each host, caches it for up to 24 hours (longer if it cannot refresh it), and ignores content after 500 KiB. (Sources: Google, List of Google's common crawlers and How Google interprets the robots.txt specification, read 26 October 2026.)

Google's user-triggered fetchers are a different group. Google says that because the end user triggers the fetch, they ignore robots.txt rules. That is why Googlebot rules do not cover, for example, Google Site Verifier.

Which robots.txt token controls Googlebot?

You need to match only one of a crawler's tokens for a rule to apply, and a more specific token wins. Google's own example: with groups for googlebot-news, * and googlebot, Googlebot News follows the googlebot-news group and Googlebot (web) follows the googlebot group. Groups for a named crawler are not combined with the * group. The general rules are in robots.txt explained.

What is the robots.txt for Googlebot?

Allow everything for Googlebot while blocking one directory for other crawlers:

User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /private/

Block a section from Googlebot only (Google's example syntax uses Allow for an exception inside a blocked path):

User-agent: Googlebot
Disallow: /archive
Allow: /archive/1Q84

When rules conflict, Google applies the most specific rule by path length, and for ties the least restrictive one. Paths are case-sensitive. Disallow: / under User-agent: Googlebot blocks the whole site from Search crawling.

Does Googlebot have anything to do with AI?

Yes, indirectly. Google says AI is built into Search, which is why robots.txt directives for Googlebot are the control for how your site is crawled for Search, including AI Overviews and AI Mode. To limit how much of a page shows in Search, Google points to nosnippet, data-nosnippet, max-snippet and noindex. To limit AI training and grounding in some of Google's other systems, it points to Google-Extended. (Source: Google, AI features and your website, read 26 October 2026.) So the choice is not symmetrical: blocking Google-Extended leaves Search and its AI features alone, while blocking Googlebot stops Search from crawling your pages. See the Googlebot and Google-Extended pages for each in short.

How do I check that a request really is Googlebot?

The user-agent string can be spoofed, Google warns. Google offers two methods. For one-off lookups, run a reverse DNS lookup on the address in your log, check that the name ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that name and confirm it returns the original address. For large scale, compare the address with Google's published JSON ranges.

Googlebot Smartphone and Desktop send these user agents (Chrome version shown as W.X.Y.Z): Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) and Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36. The IP checker and the user-agent lookup help with a single line from your log.

How often is the name Googlebot faked?

On four small sites on one server, 1,785 requests naming Googlebot between 19 and 25 October 2026 came from addresses in Google's published IP lists and 81 (4.3 percent) did not. Those 81 are requests that borrowed the name. The result describes new, small sites in one week; it does not give a rate for yours. The method and a script are in the 48-hour log study.

How do I test whether my robots.txt blocks Googlebot?

Enter a page address in the checker. It evaluates your robots.txt for 24 crawlers, Googlebot and Bingbot among them, and shows which may fetch that page. A missing robots.txt allows everything. If Googlebot is allowed but pages are still missing from Google, see why Google may not be crawling your site and the usual robots.txt mistakes.

Checked 26 October 2026

Sources are Google's pages on common crawlers, verifying Google crawlers, the robots.txt specification and AI features, all read on 26 October 2026. We paraphrase and quote short phrases. Google can change them. The request counts are our own aggregates of four small sites on one server, addresses counted and not stored.