Crawlerpeek
Check a site

archive.org_bot (Internet Archive): what it is and how to allow or block it

Updated · checked against the operator's documentation on

archive.org_bot is Reported to be the crawler the Internet Archive uses for Wayback Machine and Archive-It captures. Its robots.txt token is archive.org_bot.

Facts

Operator
Internet Archive
Category
Special-purpose crawler (reported)
robots.txt token
archive.org_bot
Honors robots.txt?
Disputed. Some directories say it respects robots.txt, others say not reliably. We do not assert either way.
User-agent example
Contains archive.org_bot (the operator documents no full example string)
User agent we logged
Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot) Zeno/7d76126 warc/v0.8.102Most common variant in our own logs, 27 requests, 19 to 25 Oct 2026. Static snapshot, not the live counter below. A logged string is a claim: anyone can send it.
Official source
Internet Archive: archive.org_bot (page has no readable text)
Compared with the source

What it is used for

Reported as the user agent of Wayback Machine and Archive-It captures. We could not read an operator policy for it.

What happens if you block it

Unknown from the operator's documentation. It is reported to be an archive crawler, so blocking it would probably affect Wayback Machine and Archive-It captures rather than search rankings; that is unconfirmed.

The page named in the user agent returned no readable text when we fetched it on 2026-10-25, so compliance is unverified.

Is it really archive.org_bot?

Anyone can send archive.org_bot as a user agent, and Internet Archive publishes no IP list or reverse-DNS method that we could find, so a log line cannot prove a request is real. Check whether it fetched /robots.txt and whether the address belongs to the operator's network.

  • We did not find an IP list or a reverse-DNS method in the official source we compared on 2026-10-25, so the user agent alone cannot show that a request is genuine. Check the operator's page for updates.
  • Paste any user agent from your logs into the user-agent lookup, or follow the scripts in Fake Googlebots and PHP scanners.

How to allow or block archive.org_bot

Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.

Allow archive.org_bot
User-agent: archive.org_bot
Allow: /
Block archive.org_bot
User-agent: archive.org_bot
Disallow: /

Does your site allow archive.org_bot today? The checker evaluates your robots.txt for this and every other bot.Check my site

What we see on our own sites

Live counter, last 30 days: no request identifying as archive.org_bot recorded yet. The counter started after the week below, so it does not cover it. Earlier log snapshot (19 to 25 Oct 2026): 30 requests identified as archive.org_bot.

Counts come from the websites we operate; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.

Questions about archive.org_bot

What is archive.org_bot?

Reported to be the crawler the Internet Archive uses for Wayback Machine and Archive-It captures. Reported as the user agent of Wayback Machine and Archive-It captures. We could not read an operator policy for it.

Is archive.org_bot safe and legitimate?

archive.org_bot is reported to be a special-purpose crawler run by Internet Archive, but we could not read an operator page that confirms it. A user-agent string is only a claim, and anyone can send it. We did not find an IP list or reverse-DNS method in the official source we compared on 2026-10-25, so the user agent alone cannot prove a request is real.

Does archive.org_bot obey robots.txt?

Disputed. Some directories say it respects robots.txt, others say not reliably. We do not assert either way.

How do I block archive.org_bot?

Robots.txt may not stop it. Some directories say it respects robots.txt, others say not reliably. We do not assert either way. If you add a group anyway, use User-agent: archive.org_bot followed by Disallow: /.

Will blocking archive.org_bot hurt my Google rankings or AI visibility?

It is an archive crawler, not a search index. Blocking it would affect Wayback Machine captures rather than your rankings, but the operator documents no policy we could read, so this is unconfirmed.

Related bots

Other Special-purpose crawlers

All Special-Purpose Crawlers · Bot directory · Which bots to allow or block · robots.txt generator · See a page as a bot