Meta crawlers: what each bot does
Meta runs 3 bots that we document: meta-externalagent, meta-externalfetcher, facebookexternalhit. They have different jobs (ai training crawler, ai user-triggered fetcher, social preview fetcher), so a robots.txt rule for one does not apply to the others.
| Bot | Category | robots.txt token | Honors robots.txt? | What blocking does |
|---|---|---|---|---|
| meta-externalagent | AI training crawler | meta-externalagent | Yes, per Meta | Blocking it with a Disallow rule in robots.txt; Meta notes changes can take up to 24 hours to take effect. |
| meta-externalfetcher | AI user-triggered fetcher | meta-externalfetcher | May bypass (user-initiated) | Meta says it may bypass robots.txt rules because the fetches are user-requested, so a Disallow rule may not block it. |
| facebookexternalhit | Social preview fetcher | facebookexternalhit | May bypass (integrity checks) | If it cannot fetch the page, shared links lose their preview card. |
Stay findable, opt out of training
If you want Meta's search and fetch bots to keep access but its training bot and control tokens to stay away, this is the robots.txt for it. It is one option, not a recommendation.
User-agent: meta-externalfetcher Allow: / User-agent: meta-externalagent Disallow: /
No request from a Meta bot has been recorded on our sites in the last 30 days. Bot traffic data