About Crawlerpeek

Crawlerpeek is a free reference and checker for web crawlers. It lists the bots that visit websites (AI, search engine, SEO tool and social preview bots), says what each one does and links the operator's own documentation, and it checks whether your site lets them in.

What you can do here

• Check a page with the AI crawler checker or see it as a bot without JavaScript.
• Build a robots.txt with the robots.txt generator.
• Look up a bot in the bot directory or by operator.
• Read the guides on robots.txt and on blocking bots on common servers.
• See our own bot traffic data, open as JSON and CSV.

How we work

Every bot entry rests on the operator's own documentation and carries the date we last compared it. If we cannot find an official source, the bot is left out or the page says so. The details, including limits, are in the methodology. The tools are free and need no account. The provider of this site is named in the imprint at the bottom of every page.