Does Meta-ExternalAgent Respect robots.txt? 2026 Log Test

Meta documents robots.txt control for meta-externalagent. On this site 128 requests made 0 robots.txt fetches and 0 hits on disallowed paths.

Published
9 min read
  • meta-externalagent
  • robots.txt
  • meta
  • ai crawler
  • user agent
  • server logs
  • crawl-delay

Meta documents that a meta-externalagent group in robots.txt controls its AI crawler, and it names only two other Meta crawlers as able to bypass the file. On crawlerpeek.com, 128 meta-externalagent requests between 19 and 31 October 2026 included 0 robots.txt fetches and 0 requests to the 3 disallowed paths. That fits compliance. It does not prove it.

As of 1 November 2026. This is one site, 13 days and mostly one burst (124 of 128 requests came within 179 seconds on 26 October), so read every number as a count, not a rate. The raw logs for that window have since been rotated out, so we could not recount these numbers on 9 November 2026.

A Meta crawl shows up in an access log like this. The user agent is worth reading twice:

203.0.113.10 - - [26/Oct/2026:01:30:49 +0000] "GET /bots/gptbot HTTP/2.0" 200 9120 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))"

The address, path and size are placeholders, and the Chrome version digits are shortened; the rest is the format we logged. It looks like Chrome, yet the token at the end is Meta's. Meta's own wording about that token comes first, because the logs only mean something next to it.

What does Meta document about robots.txt for Meta-ExternalAgent?

Meta's web crawler page lists Meta-ExternalAgent as a crawler used "for use cases such as training foundation AI models" or "improving products by indexing content directly", and tells site owners: "In order to block these crawlers, add a disallow for the relevant crawler to robots.txt." The page shows "Updated: May 21, 2026" at the top (read 9 November 2026).

What the page says about bypassing the file is specific, and Meta-ExternalAgent is not in it:

Meta crawler What Meta says it does What Meta says about robots.txt
Meta-ExternalAgent Crawls the web for training AI models or indexing content directly Block it with a Disallow for meta-externalagent; the page gives an Allow/Disallow example for that token
Meta-ExternalFetcher Fetches individual links at a user's request "may bypass robots.txt because it performs fetches that were requested by the user"
FacebookExternalHit Crawls content shared on a Meta app to build the link preview "might bypass robots.txt when performing security or integrity checks"
Meta-WebIndexer Crawls the web to improve Meta AI search results Allowing it "helps us cite and link to your content in Meta AI’s responses"; no bypass exception named

The page never writes "Meta-ExternalAgent respects robots.txt". The statement is indirect: robots.txt is the documented way to block it, and the bypass exceptions name two other crawlers. The catalogue pages for meta-externalagent, meta-externalfetcher and facebookexternalhit cover each token, and the Meta operator page ties them together. The question for your logs is narrower than "does Meta obey": did this token ask for anything your file forbids?

What did meta-externalagent do on this site from 19 to 31 October 2026?

On crawlerpeek.com, meta-externalagent made 128 requests on two days, 26 and 28 October 2026, and never fetched robots.txt under its own user agent. Every request was a GET answered with status 200. Our own test networks are excluded.

Measure (19 to 31 Oct 2026) Count
meta-externalagent requests 128
robots.txt fetches under that user agent 0
Requests to /imprint, /privacy or any URL with a query string (the paths behind the three rules) 0
Distinct paths requested 105
facebookexternalhit requests (all) 11
facebookexternalhit robots.txt fetches 7

Method and limits. Period: 19 to 31 October 2026, UTC; this site's own server logs, aggregated. The crawl came in three bursts.

Burst (UTC) Requests Distinct paths Duration
26 Oct, from 01:30:49 124 104 179 s
26 Oct, from 19:36:33 3 3 39 s
28 Oct, 09:12:19 1 1 single request

The first burst ran at about 0.69 requests per second, mostly on bot pages (61 of 128 requests went to /bots/) and guides (19). The five user-agent strings differ only in operating system, Chrome version or an Edge token; all end with meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler). The version in the token is 1.1, as on Meta's page.

Pages fetched on 1 November 2026 link to /imprint and /privacy in the footer (4 of 4 checked pages), so the crawler could have found them. We did not archive the HTML as served on 26 October, so that is a hint, not a record. Was the file even saying "no" at the time?

Did robots.txt already disallow those paths when the burst arrived?

Yes, as best we can reconstruct it: the file served on 26 October 2026 had a group naming meta-externalagent with Disallow: /imprint, Disallow: /privacy and Disallow: /*?u=. We did not archive that file, so the evidence is indirect.

This site's robots.txt is generated by code. Our archive records the response size of each fetch: 1,466 bytes for all 75 fetches on 26 October and 2,118 bytes from 30 October, so the file changed after the burst. We regenerated the file from the repository state of 25 October 2026 (committed at 22:34 UTC, about three hours before the burst began at 01:30 UTC on 26 October). That version lists 51 user-agent tokens, including meta-externalagent, meta-externalfetcher and facebookexternalhit, followed by the same three Disallow lines. It comes out at exactly 1,466 bytes, the size the archive recorded.

This is a reconstruction, not a copy; it rests on a size match and the code at one commit. The file served on 1 November 2026 (2,118 bytes, 76 user-agent lines) still has the meta-externalagent group with the same three rules. The rules are narrow: two pages and one query pattern, so "0 violations" means 0 violations of three rules.

Does the facebookexternalhit robots.txt fetch before the burst prove Meta read the file?

No. It is a timing observation. One second before the 124-request burst on 26 October, facebookexternalhit fetched robots.txt, from a different /24 network than the burst. Meta's page describes facebookexternalhit as the link-preview crawler and says nothing about passing robots.txt contents to other crawlers.

The other timings do not settle it either: the nearest earlier robots.txt fetch before the 19:36 burst on 26 October came 18.1 hours before, and before the single request on 28 October 14.9 hours before. All 7 robots.txt fetches by Meta's crawlers in the period came from facebookexternalhit, none from meta-externalagent. Meta says crawlers "may cache the contents of robots.txt for up to 24 hours", so a crawler can act on rules it fetched earlier or elsewhere. Our logs cannot show where meta-externalagent got its rules, only that its requests avoided what the file forbade. Which group should you write, then?

How do you block meta-externalagent in robots.txt?

Add a group with the token meta-externalagent, the token Meta documents, and a Disallow line. RFC 9309 says crawlers "MUST use case-insensitive matching to find the group", so Meta-ExternalAgent works too.

User-agent: meta-externalagent
Disallow: /

To keep it out of one area only, use Meta's example form: Allow: / plus Disallow: /private/.

Two details matter. RFC 9309 applies the * group only when no group matches the product token, so a named group replaces it for that bot: repeat any shared rules inside it. Blocking meta-externalagent also leaves Meta-WebIndexer alone, the crawler Meta says it uses to cite and link content in Meta AI answers, so you can refuse training-type crawling and still stay citable. And this group does not cover Meta-ExternalFetcher or facebookexternalhit, which Meta says may bypass robots.txt; stopping those takes a server rule, for example on the user-agent string. The guide to blocking AI crawlers shows server-level options, and the robots.txt basics cover group matching.

Meta says changes can take up to 24 hours to apply.

What about meta-externalagent and crawl-delay?

Meta's crawler page does not mention Crawl-delay, so Meta documents no way to slow this crawler through robots.txt. RFC 9309 does not define a Crawl-delay record either; section 2.2.4 says crawlers "MAY interpret other records that are not part of the robots.txt protocol". We did not test whether Meta reads one; the first burst here ran at 0.69 requests per second against a file with no Crawl-delay line. A rate limit in the web server is the lever you can verify yourself.

How do you check your own logs and confirm the request came from Meta?

Search the log for the token, count robots.txt fetches under it, compare the paths with your Disallow lines, then look up the address's network. These commands assume the nginx or Apache combined log format, where field 1 is the address and field 7 the path.

# 1. All requests that carry the token (case-insensitive)
grep -ci 'meta-externalagent' access.log

# 2. How many of them fetched robots.txt?
grep -i 'meta-externalagent' access.log | grep -c 'GET /robots.txt'

# 3. Did any ask for a path you disallow? (adapt the pattern to your file)
grep -i 'meta-externalagent' access.log | awk '{print $7}' | grep -E '^/(imprint|privacy)|[?]u='

# 4. How many distinct addresses?
grep -i 'meta-externalagent' access.log | awk '{print $1}' | sort -u | wc -l

We ran them on a six-line sample log we wrote ourselves in the real format (four meta-externalagent lines, among them /imprint and a query string, plus a facebookexternalhit robots.txt line and a browser line). The output was 4, 0, the two disallowed paths /imprint and /verify-bot-ip?u=1, and 2. A real violation looks like command 3 printing a line; on our logs for 19 to 31 October 2026 it printed nothing.

Meta's page is thin on verification. For FacebookExternalHit it says: "Add to your allow list either the user agent strings or the IP addresses (more secure) used by the crawler." The current version contains no IP list, no AS number and no whois instruction. A copy of the page archived on 1 May 2026 still had a "Crawler IPs" section naming AS32934 and the command whois -h whois.radb.net -- '-i origin AS32934' | grep ^route; a copy from 1 June 2026 does not. A user agent proves nothing, since anyone can send it. What we did instead is not in Meta's current documentation: we looked up the origin network of each address with the Team Cymru whois service (whois -h whois.cymru.com " -v 203.0.113.10", with your own address in place of the example). AS32934 is the number Meta's older page named.

User agent Requests Distinct addresses From AS32934 (Facebook, Inc.)
meta-externalagent 128 57 128
facebookexternalhit 11 11 7

All 128 meta-externalagent requests came from AS32934. For facebookexternalhit, 4 of 11 requests that used the name came from other networks, so the string alone would have counted them as Meta. An AS match shows the network, not the behavior, and it is weaker than a published list. For operators that publish ranges, the IP verification tool compares an address with them, as in GPTBot IP ranges.

Common follow-up questions

What is meta-externalagent?

Meta-ExternalAgent is a Meta crawler that, per Meta's web crawler page (updated 21 May 2026), crawls the web for use cases such as training foundation AI models or indexing content directly. Its user agent ends with meta-externalagent/1.1.

Is Meta-ExternalAgent ignoring my robots.txt?

Check with the commands above. In our logs, 128 requests touched none of the 3 disallowed rules. If you see one on a forbidden path, confirm the address belongs to AS32934 first: a user-agent string can be forged.

Is meta-externalagent 1.1 the same crawler as meta-externalfetcher?

No. Meta lists them as separate crawlers; the 1.1 is a version number in both strings. Meta-ExternalFetcher fetches individual links at a user's request and, per Meta, may bypass robots.txt. We saw 0 requests from it.

Meta describes link previews as the job of FacebookExternalHit, a different crawler. A meta-externalagent group does not name it, and we did not test what blocking either does to previews.

Paste your file into the robots.txt validator to check, before Meta's next visit, that the meta-externalagent group says what you intend.

Sources

Checked 1 November 2026.