AI training opt-out tracker: how the top 500 websites use robots.txt

On 31 October 2026, 2 of the 227 readable robots.txt files among the Tranco top 500 had a group for the shared AI training token __usage_ai-training (twitter.com, x.com). 68 files (30 %) named at least one AI training control: the shared token, Google-Extended, GPTBot or ClaudeBot. We repeat the count every month with the same method.

By the Crawlerpeek editorial team · Run of 2026-10-31, 19:48 to 19:57 UTC · Tranco list 746VX · first run · Method

Download CSV (one row per run)Per-domain CSV (2026-10-31)Open JSON

Latest run

31 October 2026. Shares are of the 227 readable files; 273 of the 500 domains gave no readable robots.txt.

A file counts for a token when one of its User-agent lines names it exactly, apart from case. User-agent: * never counts: the draft says wildcard groups must not be read as matching the shared token, and a group for all bots says nothing about AI training in particular.

Who uses the shared token

x.com and twitter.com served the same file, so this is one site owner.

Run by run

A new run is added at the start of each month. The list of top sites changes a little from month to month, so a change of one or two files can come from the list as well as from the sites.

What is being counted

__usage_ai-training is a policy token defined in the IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00 (30 October 2026, an individual submission). A robots.txt group that names it states one preference against or in favour of AI training for every provider that reads it, instead of one group per vendor bot. It is not a crawl directive. Until providers read it, the vendor tokens Google-Extended (Google), GPTBot (OpenAI) and ClaudeBot (Anthropic) are how sites opt out of training, so we count those too. Background, provider dates and a parser test: __usage_ai-training in robots.txt: What to Add Now.

To add the group to your own file, use the robots.txt generator; the robots.txt validator flags misspelled variants of the token.

Method

1. Sample: the top 500 domains of the Tranco list of the run day (list ID per run in the table and the files). Tranco ranks domains, not websites, so the sample includes CDN, API and tracking domains without a public site.
2. One pass: one HTTPS request for /robots.txt on the domain, and one on www. plus the domain when the first answer was not HTTP 200. Redirects are followed, certificates are checked, the timeout is 10 seconds and at most 1 MB is read.
3. Polite: at most 2 requests per second over the whole run, robots.txt files only, with a user agent that names the survey and a contact page.
4. A file is readable when the answer is HTTP 200, not empty and not an HTML page. Everything else (errors, 403, 404, timeouts, HTML) counts as not readable and is left out of the shares.
5. Each readable file is split into groups as in RFC 9309 (consecutive User-agent lines share the rules below them). A token counts when a User-agent line names it exactly, apart from case; the Disallow: / column shows whether that group closes the whole site.
6. The script and its parser are open: scripts/ai-training-optout-scan.py, Python standard library only. The per-domain file of every run lets you recount each number.

Limits

• 273 of 500 domains gave no readable file in the latest run. Many are infrastructure domains without a website; others block unknown clients, which our survey client is. The shares describe the readable files, not the top 500 as a whole.
• A named token is not always an opt-out: a group can also allow a bot. The Disallow: / column counts only groups that close the whole site; partial rules are in the per-domain file.
• robots.txt states a preference; it does not show whether a provider follows it, and it says nothing about meta tags, HTTP headers or contracts that also express AI preferences.
• The top 500 are large sites. Small and medium sites may behave very differently.

Columns

The per-domain file has one row per domain and run:

The run file (ai-training-opt-out.csv) has one row per run with the counts above: scan_date, tranco_list_id, sample, readable, token_group, token_group_pct_readable, any_ai_training_control, any_ai_training_control_pct_readable, google_extended_named, google_extended_blocks_root, gptbot_named, gptbot_blocks_root, claudebot_named, claudebot_blocks_root, all_three_named, content_signal, token_typo, token_string_outside_group, period_start, period_end, per_domain_file. Percentages are of readable files. Both files quote every field and have no comment lines.

Crawlerpeek. (2026). AI training opt-out tracker: robots.txt of the Tranco top 500 (Version 2026-10-31) [Data set]. https://crawlerpeek.com/data/ai-training-opt-out

Changelog

• 2026-10-31: First measurement, the day after the IETF draft that defines the __usage_ai-training token (30 October 2026). Top 500 rows of the Tranco list 746VX, identical to that day's daily file.

Related: robots.txt of 151 big sites · block AI crawlers · AI training crawlers · bot traffic data