Search engine crawlers: Googlebot, Bingbot, Applebot and more
Search engine crawlers fetch pages for a search index. Blocking them removes you from that search engine and from the AI features built on its index.
A traditional search engine crawler. AI features of the same company often draw on its index.
3 bots in this group
| Bot | Owner | robots.txt token | Honors robots.txt? | Requests on our sites (30 days) |
|---|---|---|---|---|
| Googlebot | Googlebot | Yes, per Google | 26 | |
| Bingbot | Microsoft | Bingbot | Yes, per Bing | 28 |
| Applebot | Apple | Applebot | Yes, per Apple | none yet |
54 requests in this group on our sites in the last 30 days.
Block the whole group
Paste into robots.txt to ask every bot in this group to stay away. Only do this if you mean it: read what blocking changes first.
User-agent: Googlebot Disallow: / User-agent: Bingbot Disallow: / User-agent: Applebot Disallow: /