AI bot list: download as CSV or JSON

Name: AI bot reference list
Creator: Crawlerpeek
License: https://creativecommons.org/licenses/by/4.0/

24 bots in 7 groups, one row each: operator, purpose, user agent, robots.txt token and what the operator says about robots.txt, with a link to the official source and the date we last compared the entry with it (version 2026-10-18).

Download CSV Open JSON

Cite this list

If you use the list in an article, a README or a tool, please credit it and link to crawlerpeek.com/data/ai-bots.

Citation (APA style)

Crawlerpeek. (2026). AI bot reference list: operators, purposes, user agents and robots.txt tokens (Version 2026-10-18) [Data set]. https://crawlerpeek.com/data/ai-bots

Attribution for a web page (HTML)

Data: <a href="https://crawlerpeek.com/data/ai-bots">Crawlerpeek AI bot reference list</a> (CC BY 4.0)

License: our compilation and descriptions are CC BY 4.0. Bot names, user-agent tokens and the linked official pages belong to their operators and are not covered by this license.

Columns

Columns of the AI bot list
Column	Meaning
`slug`	Stable identifier; also the path of the bot page
`name`	Bot name as the operator writes it
`operator`	Company that runs the bot
`purpose`	Machine-friendly purpose: ai_training, ai_search, ai_user_request, ai_control_token, search_engine, seo_tool or social_preview
`category`	The same purpose as a readable label
`robots_token`	The name to use in a robots.txt User-agent line
`user_agent_contains`	Text that appears in the bot's user-agent header (empty for control tokens that never send requests)
`honours_robots_txt`	What the operator says, in a few words; details on the bot page
`official_source`	The operator's own documentation we compared the entry with
`ip_list_url`	Machine-readable list of the operator's IP ranges, where one is published
`last_checked`	Date (UTC) on which we compared the entry with its official source
`page`	Our page for the bot

Limits

A user agent is a claim: anyone can send any user-agent string, so the list says what a bot calls itself, not who is behind a request.
The list is small and checked by hand against operator documentation. It is not complete, and operators change their bots without notice. Corrections are welcome via the contact form.
The CSV has no comment lines and quotes every field, so spreadsheets and data tools import it unchanged. The license is stated here, in the JSON file and in the HTTP Link header.

Browse the same data as pages in the bot directory, or read how we check in the methodology.