Crawlerpeek
Check a site

meta-webindexer (Meta): what it is and how to allow or block it

Updated · checked against the operator's documentation on

meta-webindexer is the crawler Meta uses to improve the quality of Meta AI search results. Meta says allowing it in robots.txt helps Meta cite and link to your content in Meta AI's responses. Block it with User-agent: meta-webindexer. Its robots.txt token is meta-webindexer.

Seen meta-webindexer/1.1 in your logs?

meta-webindexer/1.1

The example string from Meta's documentation. Version numbers change; the token meta-webindexer stays.

What it is
An AI search crawler run by Meta.
Honors robots.txt
YesYes, per Meta.
Real or fake?
No official checkThe user agent is only a claim, and Meta publishes no IP list or DNS check we could find.
If you block it
Per Meta, allowing it in robots.txt helps Meta cite and link to your content in Meta AI's responses, so a Disallow rule works against that.
Block or allow itLook up another user agent

Facts

Operator
Meta
Category
AI search crawler
robots.txt token
meta-webindexer
Honors robots.txt?
Yes, per Meta. Meta says to block its crawlers by adding a disallow for the relevant crawler to robots.txt, and to allow up to 24 hours for changes to apply. Meta lists a robots.txt bypass only for Meta-ExternalFetcher (user-requested fetches) and FacebookExternalHit (security and integrity checks), not for this crawler.
User-agent example
meta-webindexer/1.1
Official source
Meta: Web crawlers
Compared with the source

What it is used for

Per Meta, it navigates the web to improve Meta AI search result quality for users, and in doing so analyzes online content to enhance the relevance and accuracy of Meta AI.

Builds the index an AI assistant searches when it answers. Blocking it can keep you out of that assistant's answers.

What happens if you block it

Per Meta, allowing it in robots.txt helps Meta cite and link to your content in Meta AI's responses, so a Disallow rule works against that. Meta-ExternalAgent, its training and indexing crawler, has its own token.

Meta's crawler page (updated 21 May 2026) names no IP list, ASN or reverse-DNS method for this crawler, so the user agent alone cannot show that a request is genuine.

Is it really meta-webindexer?

Anyone can send meta-webindexer as a user agent, and Meta publishes no IP list or reverse-DNS method that we could find, so a log line cannot prove a request is real. Check whether it fetched /robots.txt and whether the address belongs to the operator's network.

How to allow or block meta-webindexer

Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.

Allow meta-webindexer
User-agent: meta-webindexer
Allow: /
Block meta-webindexer
User-agent: meta-webindexer
Disallow: /

Block meta-webindexer at the server

robots.txt asks; a server rule refuses. These answer 403 to any request whose user agent contains meta-webindexer (case-insensitive). They stop only bots that name themselves honestly, so keep the robots.txt group as well.

nginx
# server { ... }
if ($http_user_agent ~* "meta-webindexer") {
    return 403;
}
Apache (.htaccess)
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} meta-webindexer [NC]
RewriteRule .* - [F,L]

Step by step, with a test command: Block AI bots in nginx by user agent · Block AI bots in Apache with .htaccess · Block AI crawlers on Cloudflare · Block AI bots on Vercel

Does your site allow meta-webindexer today? The checker evaluates your robots.txt for this and every other bot.Check my site

What we see on this site

No request identifying as meta-webindexer has been recorded since 19 Oct, so there is nothing to show yet. We never fill this in with estimates.

Counts come from this site's own server; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.

Questions about meta-webindexer

What is meta-webindexer?

Meta's crawler that navigates the web to improve Meta AI search result quality for users. Per Meta, it navigates the web to improve Meta AI search result quality for users, and in doing so analyzes online content to enhance the relevance and accuracy of Meta AI.

Is meta-webindexer safe and legitimate?

The real meta-webindexer is an AI search crawler run by Meta. A user-agent string is only a claim, though, so anyone can send it. We did not find an IP list or reverse-DNS method in the official source we compared on 2026-11-02, so the user agent alone cannot prove a request is real.

Does meta-webindexer obey robots.txt?

Yes, per Meta. Meta says to block its crawlers by adding a disallow for the relevant crawler to robots.txt, and to allow up to 24 hours for changes to apply. Meta lists a robots.txt bypass only for Meta-ExternalFetcher (user-requested fetches) and FacebookExternalHit (security and integrity checks), not for this crawler.

How do I block meta-webindexer?

Add a group to the robots.txt at the root of your domain: User-agent: meta-webindexer followed by Disallow: /. A group that names the bot overrides a general User-agent: * group.

Will blocking meta-webindexer hurt my Google rankings or AI visibility?

It can. This bot builds the index an AI assistant searches when it answers, so blocking it can keep you out of that assistant's answers. Per Meta, allowing it in robots.txt helps Meta cite and link to your content in Meta AI's responses, so a Disallow rule works against that. Meta-ExternalAgent, its training and indexing crawler, has its own token.

What does "meta-webindexer/1.1" mean in my logs?

It is the user-agent string in Meta's documentation for meta-webindexer, an AI search crawler run by Meta. The part that robots.txt rules match is the token meta-webindexer. The string is only a claim, because anyone can send it; the section "Is it really meta-webindexer?" above shows how to check a request.

Related bots

More from Meta

Other AI search crawlers

All AI Search Crawlers · Bot directory · Bots from Meta · Which bots to allow or block · robots.txt generator · See a page as a bot