Methodology: how we check bots, with sources and limits
Crawlerpeek is a free tool that shows whether AI bots can read a web page. It works without an account and shows its method and its sources, including when each was last compared with the original.
What the checker does
- It requests your page like a browser would, without running JavaScript, and reads title, description, headline, text, links, canonical and noindex from the HTML.
- It requests
/robots.txtand evaluates it for each bot we track with the longest-match rule of RFC 9309. A missing robots.txt counts as allowed. - It requests
/llms.txtand checks that it is a text file in the documented format. - That is three requests (up to five when one is retried), each limited to 5 seconds. The page is read up to 1 MB, robots.txt and llms.txt up to 500 KB. Private and local addresses are refused. The robots.txt and llms.txt requests identify as
CrawlerpeekBot/1.0. - It does not send requests with other companies' bot user agents, does not run JavaScript and does not log in anywhere.
What it cannot tell you
- Whether a firewall or CDN blocks a bot that robots.txt allows.
- Whether a bot has crawled you, or whether an assistant will mention you. Allowed is not the same as cited.
- What hosts that serve prerendered pages to verified crawlers show those crawlers.
Sources and check dates
Bot facts come from the operators' own documentation. We last compared all entries on . Where an operator documents nothing (Bytespider) or its page could not be re-read (Bing), the bot page says so.
| Source | Covers |
|---|---|
| OpenAI: Overview of OpenAI crawlers | GPTBot, OAI-SearchBot, ChatGPT-User |
| Anthropic: Does Anthropic crawl data from the web? | ClaudeBot, Claude-SearchBot, Claude-User |
| Perplexity: Bots and user agents | PerplexityBot, Perplexity-User |
| Google: Common crawlers | Googlebot, Google-Extended |
| Google: AI features and your website | No special files needed |
| Apple: About Applebot | Applebot, Applebot-Extended |
| Common Crawl: CCBot | CCBot |
| Meta: Web crawlers | meta-externalagent, meta-externalfetcher |
| Bing Webmaster: Which crawlers does Bing use? | Bingbot (page needs JavaScript; re-check by hand) |
| llmstxt.org | The llms.txt proposal |
| Ahrefs: llms.txt study | 137,210 domains, May 2026 |
Bot pages: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, Bytespider, Googlebot, Bingbot, Applebot, meta-externalfetcher, DuckAssistBot, Amazonbot, AhrefsBot, SemrushBot, MJ12bot, facebookexternalhit, Slackbot.
How the bot traffic numbers are made
When a request reaches one of our sites and its user-agent header contains the documented token of a bot we track, a counter for that day, site, bot and page type goes up by one. We store nothing else: no IP address, no user-agent string, no URL. A user agent can be faked, so the numbers are requests that identified as a bot. Results are on the data page and, per bot, on each bot page.
Corrections
If a bot changed its token or a source moved, write to hello@crawlerpeek.com and we will check it against the original.