Crawlerpeek
Check a site

AgentTrustBot (agenttru.st): what it is and how to allow or block it

Updated · checked against the operator's documentation on

AgentTrustBot is a crawler that looks for published AI agent discovery documents on newly seen hostnames. Its robots.txt token is AgentTrustBot.

Facts

Operator
agenttru.st
Category
Special-purpose crawler
robots.txt token
AgentTrustBot
Honors robots.txt?
Yes, per agenttru.st. The operator says it honors robots.txt per hostname and port, backs off on 429 and 503 with Retry-After, and offers removal at its delist page. The token is case-insensitive; the older name agenttrust-collector is still honored.
User-agent example
Contains agenttrustbot (the operator documents no full example string)
User agent we logged
AgentTrustBot/1.0 (id=95ecd7a1d908; +https://agenttru.st/crawler; purpose=AI agent directory; contact=admin@agenttru.st)Most common variant in our own logs, 6 requests, 19 to 25 Oct 2026. Static snapshot, not the live counter below. A logged string is a claim: anyone can send it.
Official source
agenttru.st: crawler information
Compared with the source

What it is used for

Per the operator, it reads hostnames from public Certificate Transparency logs, checks each for agent discovery documents and lists the agents it finds in a directory.

Not the crawler behind a search index. What blocking it changes is specific to its tool, so read the next section.

What happens if you block it

Your hostnames are left out of the agent directory, so any agent discovery documents on them are not listed.

The user agent carries an id that changes with each visit, so match the token only. The operator's page lists its addresses inline and does not offer a machine-readable list.

Is it really AgentTrustBot?

Anyone can send AgentTrustBot as a user agent, so a log line proves nothing by itself. The way to tell is the requesting IP address.

How to allow or block AgentTrustBot

Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.

Allow AgentTrustBot
User-agent: AgentTrustBot
Allow: /
Block AgentTrustBot
User-agent: AgentTrustBot
Disallow: /

Does your site allow AgentTrustBot today? The checker evaluates your robots.txt for this and every other bot.Check my site

What we see on our own sites

Live counter, last 30 days: no request identifying as AgentTrustBot recorded yet. The counter started after the week below, so it does not cover it. Earlier log snapshot (19 to 25 Oct 2026): 6 requests identified as AgentTrustBot.

Counts come from the websites we operate; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.

Questions about AgentTrustBot

What is AgentTrustBot?

A crawler that looks for published AI agent discovery documents on newly seen hostnames. Per the operator, it reads hostnames from public Certificate Transparency logs, checks each for agent discovery documents and lists the agents it finds in a directory.

Is AgentTrustBot safe and legitimate?

The real AgentTrustBot is a special-purpose crawler run by agenttru.st. A user-agent string is only a claim, though, so anyone can send it. Compare the requesting IP address with the published IP ranges. agenttru.st documents verification by a reverse lookup followed by a forward lookup, with names under agenttru.st.

Does AgentTrustBot obey robots.txt?

Yes, per agenttru.st. The operator says it honors robots.txt per hostname and port, backs off on 429 and 503 with Retry-After, and offers removal at its delist page. The token is case-insensitive; the older name agenttrust-collector is still honored.

How do I block AgentTrustBot?

Add a group to the robots.txt at the root of your domain: User-agent: AgentTrustBot followed by Disallow: /. A group that names the bot overrides a general User-agent: * group.

Will blocking AgentTrustBot hurt my Google rankings or AI visibility?

This is not the crawler behind a search index, so blocking it is not a ranking decision. Your hostnames are left out of the agent directory, so any agent discovery documents on them are not listed.

Related bots

Other Special-purpose crawlers

All Special-Purpose Crawlers · Bot directory · Which bots to allow or block · robots.txt generator · See a page as a bot