Amazonbot (Amazon): what it is and how to allow or block it
Amazonbot is Amazon's crawler, used to improve its products and services and possibly to train Amazon AI models. Its robots.txt token is Amazonbot.
Facts
- Operator
- Amazon
- Category
- AI training crawler
- robots.txt token
Amazonbot
- Honors robots.txt?
- Yes, per Amazon. Respects robots.txt; crawl-delay is not supported. It also honors page-level noindex and none, noarchive (do not use for model training), and rel=nofollow.
- User-agent example
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36
- Official source
- Amazon: Amazonbot
- Published IP ranges
- https://developer.amazon.com/amazonbot/ip-addresses/
- Compared with the source
What it is used for
Amazon says it uses Amazonbot to improve its products and services, and that content may be used to train Amazon AI models. Collects public pages that may be used to train AI models. Blocking it is a statement about training, not about being found.
What happens if you block it
Amazon documents that Amazonbot respects the Robots Exclusion Protocol. It reads your robots.txt or a copy cached for up to 30 days; changes can take about 24 hours.
Amazon's page does not name a separate robots.txt token; rules addressed to Amazonbot are the documented way to target it.
How to allow or block Amazonbot
Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.
User-agent: Amazonbot Allow: /
User-agent: Amazonbot Disallow: /
Does your site allow Amazonbot today? The checker evaluates your robots.txt for this and every other bot.Check my site
What we see on our own sites
No request identifying as Amazonbot has been recorded in the last 30 days, so there is nothing to show yet. We never fill this in with estimates.
Counts come from the websites we operate; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.
Related bots
Other AI training crawlers
- GPTBot OpenAIOpenAI's crawler for content that may be used to train its generative AI models.
- ClaudeBot AnthropicAnthropic's crawler that collects web content which could contribute to training its generative AI models.
- CCBot Common CrawlThe crawler of Common Crawl, a non-profit that maintains an open repository of web crawl data.
- meta-externalagent MetaMeta's crawler that gathers web content to train AI models or to improve products by indexing.
All AI Training Crawlers · Bot directory · Which bots to allow or block · robots.txt generator · See a page as a bot