Special-purpose crawlers: testing tools, surveys, directories and archives
Special-purpose crawlers are bots that are not the crawler behind a search index: they serve a testing tool, a survey, a directory or an archive. What blocking one changes depends on that tool, not on your search rankings.
Not the crawler behind a search index. What blocking it changes is specific to its tool, so read the next section.
5 bots in this group
| Bot | Owner | robots.txt token | Honors robots.txt? | Requests on our sites (30 days) |
|---|---|---|---|---|
| GoogleOther | GoogleOther | Yes, per Google | 6 | |
| Google-InspectionTool | Google-InspectionTool | Yes, per Google | none yet | |
| archive.org_bot | Internet Archive | archive.org_bot | Disputed | none yet |
| WebMCPIndexBot | WebMCP Shield | WebMCPIndexBot | Yes, per WebMCP Shield | none yet |
| AgentTrustBot | agenttru.st | AgentTrustBot | Yes, per agenttru.st | none yet |
6 requests in this group on our sites in the last 30 days.
Block the whole group
Paste into robots.txt to ask every bot in this group to stay away. Only do this if you mean it: read what blocking changes first.
User-agent: GoogleOther Disallow: / User-agent: Google-InspectionTool Disallow: / User-agent: archive.org_bot Disallow: / User-agent: WebMCPIndexBot Disallow: / User-agent: AgentTrustBot Disallow: /