meta-externalagent (Meta): what it is and how to allow or block it
meta-externalagent is Meta's crawler that gathers web content to train AI models or to improve products by indexing. Its robots.txt token is meta-externalagent.
Facts
- Operator
- Meta
- Category
- AI training crawler
- robots.txt token
meta-externalagent
- Honors robots.txt?
- Yes, per Meta. Meta does not list it among the crawlers that may bypass robots.txt. A sibling, meta-externalfetcher, fetches single URLs on a user's request and may bypass robots.txt.
- User-agent example
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
- Official source
- Meta: Web crawlers
- Compared with the source
What it is used for
Gathers web content to train foundation AI models or to improve products by indexing content directly. Collects public pages that may be used to train AI models. Blocking it is a statement about training, not about being found.
What happens if you block it
Blocking it with a Disallow rule in robots.txt; Meta notes changes can take up to 24 hours to take effect.
How to allow or block meta-externalagent
Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.
User-agent: meta-externalagent Allow: /
User-agent: meta-externalagent Disallow: /
Does your site allow meta-externalagent today? The checker evaluates your robots.txt for this and every other bot.Check my site
What we see on our own sites
No request identifying as meta-externalagent has been recorded in the last 30 days, so there is nothing to show yet. We never fill this in with estimates.
Counts come from the websites we operate; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.
Related bots
More from Meta
- meta-externalfetcher AI user-triggered fetcherMeta's fetcher that retrieves individual URLs when a user asks, for example to help an AI complete a task.
- facebookexternalhit Social preview fetcherMeta's fetcher that loads a page when its link is shared on Facebook, Instagram or Messenger, to build the preview.
Other AI training crawlers
- GPTBot OpenAIOpenAI's crawler for content that may be used to train its generative AI models.
- ClaudeBot AnthropicAnthropic's crawler that collects web content which could contribute to training its generative AI models.
- CCBot Common CrawlThe crawler of Common Crawl, a non-profit that maintains an open repository of web crawl data.
- Bytespider ByteDanceByteDance's crawler. We found no official ByteDance documentation page for it.
All AI Training Crawlers · Bot directory · Bots from Meta · Which bots to allow or block · robots.txt generator · See a page as a bot