Googlebot (Google): what it is and how to allow or block it

Googlebot is Google's main crawler for Google Search. Google's AI features in Search draw on the same index. Its robots.txt token is Googlebot.

Facts

Operator: Google

Category: Search engine crawler

robots.txt token: Googlebot

Honors robots.txt?: Yes, per Google. Google lists Googlebot among the common crawlers that always respect robots.txt for automatic crawls.

User-agent example: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Official source: Google: Common crawlers

Compared with the source: 2026-10-18

What it is used for

Crawls pages for Google Search, including Discover and all Search features. A traditional search engine crawler. AI features of the same company often draw on its index.

What happens if you block it

Blocking Googlebot affects Google Search, including its AI features. It is the opposite of what most sites want.

How to allow or block Googlebot

Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.

Allow Googlebot

User-agent: Googlebot
Allow: /

Block Googlebot

User-agent: Googlebot
Disallow: /

Does your site allow Googlebot today? The checker evaluates your robots.txt for this and every other bot.Check my site

What we see on our own sites

In the last 30 days, 26 requests on 1 day identified as Googlebot (latest: 2026-10-19). That is a small sample: read it as an example, not as a measurement of the web.

Sitemaps: 10
robots.txt: 7
Home page: 4
Content pages: 3
Images and files: 2

Counts come from the websites we operate; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.

Other Search engine crawlers

All Search Engine Crawlers · Bot directory · Bots from Google · Which bots to allow or block · robots.txt generator · See a page as a bot