Crawler changelog
What changed at the AI and search crawlers we track: new user-agent versions, new self-identified crawlers and changes to operators' own documentation. Every entry is dated, written by a reviewer and links the operator's source.
Subscribe: Atom feed · JSON · CSV (CC BY 4.0)
No reviewed entries yet. Automated checks run every week. An entry appears here only after a reviewer has confirmed a change against the operator's own page. Nothing on this page is filled in.
How entries are made
- User-agent versions. We scan the user agents that known bots send to the sites we operate. A new version string becomes a candidate only after repeated requests on several days and, where the operator publishes an IP list, only when requests from inside that list carry it. Version strings are claims; a candidate is never an entry by itself.
- New crawlers. A user agent that names itself a bot, is not in our bot directory and keeps coming back is a candidate. It is marked unverified.
- Documentation. Once a week we read the operators' own crawler pages and compare them with the last copy. Small edits, dates and layout changes are ignored.
- Review. A person or an automated reviewer checks each candidate against the operator's page and writes the entry in plain text. Candidates that cannot be confirmed are dropped. Entries therefore appear some days after the change.
Related: fake bot study · bot directory · methodology