Crawlerpeek
Check a site

How to Verify Googlebot: DNS and IP Check

Check if a request claiming to be Googlebot is real: reverse and forward DNS, Google's published IP ranges, commands with real output, and a log example.

Published
7 min read
  • googlebot
  • bot-verification
  • reverse-dns
  • ip-ranges
  • access-logs

To verify Googlebot, resolve the request IP to a hostname, confirm the domain is googlebot.com, google.com or googleusercontent.com, then resolve that hostname forward and check it returns the same IP. You can also match the IP against the 317 prefixes in Google's common-crawlers.json (read 30 October 2026), which needs no DNS at all. Skip the check and a /wp-login.php probe that sends a Googlebot header looks like Google in your logs.

As of 30 October 2026.

Why can't you trust the Googlebot user agent?

A Googlebot user agent is a text string that any client can send, so it identifies nothing by itself. Google's verification documentation (checked 30 October 2026) names the risk directly: the check is "useful if you're concerned that spammers or other troublemakers are accessing your site while claiming to be from Google."

Here is what that looks like in a log. This line is invented for this article and uses a documentation-range address:

203.0.113.10 - - [30/Oct/2026:10:00:05 +0000] "GET /wp-login.php HTTP/1.1" 404 150 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

The header says Googlebot. The request asks for a WordPress login page. If you rate-limit by user agent, trust it in your crawl stats or exempt it from a firewall rule, you have just given a stranger Google's standing.

Google's overview page lists three signals of a Google crawler: the user-agent header, the source IP address and the reverse DNS hostname of the source IP. The user-agent header is a client-chosen string. So how do you test them for one address?

How do you check one IP with reverse and forward DNS?

The DNS check has two lookups. A reverse lookup of the IP must return a hostname under googlebot.com, google.com or googleusercontent.com, and a forward lookup of that hostname must return the original IP. Google's page requires that the forward result match the original accessing IP address.

Google's verify page (checked 30 October 2026) says its command-line method is for one-off lookups: "This method is sufficient for most use cases." Its last step is to confirm the forward result is the same as the original accessing IP address from your logs.

The host binary was not installed in the environment where we ran this check, so we ran the same lookups with Python's socket module on 30 October 2026. The reverse and forward results for Google's own example address, and the result for a documentation-range address:

reverse 66.249.66.1 -> crawl-66-249-66-1.googlebot.com
forward crawl-66-249-66-1.googlebot.com -> ['66.249.66.1']
reverse 203.0.113.10 -> ERROR herror: [Errno 1] Unknown host

The first two lines match Google's Example 1, where host 66.249.66.1 returns crawl-66-249-66-1.googlebot.com and the forward lookup returns 66.249.66.1. The third address has no reverse record, so it fails at step one.

Hostnames follow patterns that depend on the crawler. For common crawlers Google documents crawl-***-***-***-***.googlebot.com and geo-crawl-***-***-***-***.geo.googlebot.com.

DNS works for one address, but it costs two lookups per IP and a log has thousands. The published ranges avoid that.

How do you check an IP against Google's published ranges?

Google publishes the IP ranges of its common crawlers, Googlebot included, as a JSON file of CIDR prefixes named common-crawlers.json. Matching an address against that file takes no DNS lookups and scales to a whole log. Google's verify page recommends this for large-scale lookups.

On 30 October 2026 the file had a creationTime of 2026-10-30T15:46:20.000000 and held 317 prefixes: 170 IPv4 and 147 IPv6. Google's page says the addresses in the JSON files are represented in CIDR format. It states no refresh interval for the files (page read 30 October 2026), so fetch a fresh copy on a schedule you choose rather than caching it for months.

One URL detail matters if you have an old script. On 30 October 2026 https://developers.google.com/search/apis/ipranges/googlebot.json answered HTTP 301 with a Location header pointing to https://developers.google.com/static/crawling/ipranges/common-crawlers.json. Update the URL; do not depend on the redirect.

So what does this look like when you run it over a log file?

How do you verify every Googlebot request in an access log? A tested run

Extract the unique IPs from lines with a Googlebot user agent, match each against the range file, and count three outcomes: inside, outside and invalid. This script does that with Python's ipaddress module. We ran it on 30 October 2026.

Method box (evidence). Period: 30 October 2026. Input: an invented 8-line sample log, with documentation-range addresses, two Google example addresses and one IPv6 address inside Google's range. Counted: unique IPs on lines containing "Googlebot", matched against the 317 prefixes. Limits: the sample is invented, so it shows the method, not how common fake Googlebots are. This site's own Googlebot counts are not part of this article.

import ipaddress, json, sys

# usage: python3 -I verify_log.py common-crawlers.json access.log
nets = []
for p in json.load(open(sys.argv[1]))["prefixes"]:
    nets.append(ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")))

claimed = {}
for line in open(sys.argv[2]):
    if "Googlebot" in line:
        ip = line.split()[0]
        claimed[ip] = claimed.get(ip, 0) + 1

counts = {"inside": 0, "outside": 0, "invalid": 0}
for ip, n in sorted(claimed.items()):
    try:
        addr = ipaddress.ip_address(ip)
    except ValueError:
        verdict = "invalid"
    else:
        verdict = "inside" if any(addr in n_ for n_ in nets if n_.version == addr.version) else "outside"
    counts[verdict] += 1
    print(f"{ip}\t{n} requests\t{verdict}")
print(f"unique IPs: {len(claimed)}  inside={counts['inside']} outside={counts['outside']} invalid={counts['invalid']}")

Output (the script prints all five addresses in full; one of Google's example addresses is shown as a placeholder here):

$ python3 -I verify_log.py common-crawlers.json sample.log
2001:4860:4801:10::5	1 requests	inside
2001:db8::1	1 requests	outside
203.0.113.10	2 requests	outside
<google-example-ip>	1 requests	inside
66.249.66.1	2 requests	inside
unique IPs: 5  inside=3 outside=2 invalid=0

Five unique addresses claimed Googlebot. Three sit inside the ranges. 203.0.113.10 made two requests and sits outside, as does 2001:db8::1. The script matches IPv4 against IPv4 prefixes and IPv6 against IPv6, which is why the one IPv6 address inside Google's range passes.

The two that failed are the harder case. What do you do with them?

What should you do when a Googlebot request fails verification?

A failed check means the request cannot be attributed to Google's common crawlers. It does not prove malice. Before you block, check whether the request came from a different Google category, because Google uses separate hostname patterns and range files for each.

Google's verify page (checked 30 October 2026) splits its requests into categories:

Category Reverse DNS mask Range file robots.txt
Common crawlers (such as Googlebot) crawl-***-***-***-***.googlebot.com common-crawlers.json "They always respect robots.txt rules for automatic crawls."
Special-case crawlers rate-limited-proxy-***-***-***-***.google.com special-crawlers.json "may or may not respect robots.txt rules"
User-triggered fetchers google-proxy-***-***-***-***.google.com user-triggered-fetchers-google.json (one of its files) "these fetchers ignore robots.txt rules"

Special-case crawlers are used, per Google's overview page, where there is an agreement between the crawled site and the Google product about the crawl. For other Google addresses that may reach your site, Google says to match against its general list of Google IP addresses.

A working order for a failed request:

  1. Run the reverse and forward DNS check. A hostname under another Google pattern points to another category, not to a fake.
  2. Re-fetch common-crawlers.json and match again, since the verify page gives no refresh interval.
  3. If both fail, treat the source as unverified: rate-limit or block it, and leave the verified Googlebot addresses alone.

Verify the IP, never the header. For a ready answer on a single address, paste it into the IP checker. To see which bot a user-agent string names before you test it, use the user agent lookup. For what Googlebot is, read the Googlebot profile; for how scanners pose as crawlers in logs, see fake crawlers in server logs.

Common follow-up questions

Does Googlebot always respect robots.txt?

Google's verify page (checked 30 October 2026) says common crawlers such as Googlebot "always respect robots.txt rules for automatic crawls." User-triggered fetchers ignore it, because a user requested the fetch, and special-case crawlers may or may not respect it. Rules for Googlebot are covered in Googlebot and robots.txt.

Which IP ranges does Googlebot use?

Googlebot's addresses are listed in Google's common-crawlers.json. On 30 October 2026 it held 317 prefixes, 170 IPv4 and 147 IPv6. Special-case crawlers and user-triggered fetchers use other files, so a Google address missing from this one may belong to another category.

Is a request from another Google IP address Googlebot?

Not by this check. Google's page says that for other Google addresses from which your site may be accessed, such as Apps Scripts, you match against its general list of Google IP addresses. An address on that list but not in common-crawlers.json is not a verified common crawler.

Can I just block every Googlebot request that fails verification?

A failed request cannot be tied to Google's common crawlers, but Google documents other categories with their own hostnames and range files. Check those first and re-fetch the range file, then block. Blocking only after those steps keeps verified Googlebot crawling intact. Verify the IP, never the header.

Sources

Checked 30 October 2026.