meta-externalfetcher (Meta): what it is and how to allow or block it
meta-externalfetcher is Meta's fetcher that retrieves individual URLs when a user asks, for example to help an AI complete a task. Its robots.txt token is meta-externalfetcher.
Facts
- Operator
- Meta
- Category
- AI user-triggered fetcher
- robots.txt token
meta-externalfetcher
- Honors robots.txt?
- May bypass (user-initiated). May bypass robots.txt according to Meta. Its sibling meta-externalagent is the training crawler.
- User-agent example
meta-externalfetcher/1.1
- Official source
- Meta: Web crawlers
- Compared with the source
What it is used for
Retrieves individual URLs when a user requests them. Meta says it supports evaluating and improving agentic AI, including helping AI complete user tasks on websites. Opens one page because a person (or an answer being generated) needs it. The operator may treat robots.txt differently for these requests.
What happens if you block it
Meta says it may bypass robots.txt rules because the fetches are user-requested, so a Disallow rule may not block it.
How to allow or block meta-externalfetcher
Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.
User-agent: meta-externalfetcher Allow: /
User-agent: meta-externalfetcher Disallow: /
Does your site allow meta-externalfetcher today? The checker evaluates your robots.txt for this and every other bot.Check my site
What we see on our own sites
No request identifying as meta-externalfetcher has been recorded in the last 30 days, so there is nothing to show yet. We never fill this in with estimates.
Counts come from the websites we operate; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.
Related bots
More from Meta
- meta-externalagent AI training crawlerMeta's crawler that gathers web content to train AI models or to improve products by indexing.
- facebookexternalhit Social preview fetcherMeta's fetcher that loads a page when its link is shared on Facebook, Instagram or Messenger, to build the preview.
Other AI user-triggered fetchers
- ChatGPT-User OpenAIThe fetcher ChatGPT uses when a person asks it to open or look at a web page.
- Claude-User AnthropicThe agent Claude uses to open a page when a person asks it a question that needs a website.
- Perplexity-User PerplexityThe fetcher Perplexity uses to visit a page when a person's question needs it.
- DuckAssistBot DuckDuckGoDuckDuckGo's crawler that fetches pages in real time for its AI-assisted answers.
All AI User-Triggered Fetchers · Bot directory · Bots from Meta · Which bots to allow or block · robots.txt generator · See a page as a bot