AI control tokens: robots.txt names that never appear in logs
AI control tokens are names you can use in robots.txt to tell a company how crawled content may be used. They are not separate crawlers, so they never show up in your server logs.
Not a crawler you will see in your logs. A token in robots.txt that tells the operator how already collected content may be used.
2 bots in this group
| Bot | Owner | robots.txt token | Honors robots.txt? | Requests on our sites (30 days) |
|---|---|---|---|---|
| Google-Extended | Google-Extended | Yes (control token) | n/a | |
| Applebot-Extended | Apple | Applebot-Extended | Yes (control token) | n/a |
No request from this group has been recorded on our sites in the last 30 days.
Block the whole group
Paste into robots.txt to ask every bot in this group to stay away. Only do this if you mean it: read what blocking changes first.
User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: /