Crawlerpeek
Check a site

__usage_ai-training in robots.txt: What to Add Now

One of the top 500 sites used the new shared robots.txt token for AI training on 31 Oct 2026. Start dates, a parser test and the group to add.

Published
9 min read
  • robots-txt
  • ai-training
  • ietf
  • aipref
  • google-extended
  • gptbot
  • claudebot

On 31 Oct 2026, one day after Google, Microsoft and Anthropic backed one shared robots.txt opt-out for AI training, one of the top 500 sites had a __usage_ai-training group in its robots.txt: X, whose file is served at both x.com and twitter.com. The token is an ordinary User-agent group that a * group never covers. Microsoft starts reading it on 1 February 2027, Google in early 2027.

As of 31 Oct 2026.

You open your robots.txt after reading the announcement. It already blocks GPTBot, ClaudeBot and Google-Extended, one group each. Do you need a fourth group with a strange name, does it replace the other three, and does anything read it yet? The answers are in the draft itself, in the providers' own posts and in a count of what the 500 biggest sites did on day two.

What is the __usage_ai-training token, and who reads it?

__usage_ai-training is a policy token defined in an IETF Internet-Draft, draft-kelley-aipref-training-preference-signal-00. Kevin Kelley of Anthropic submitted it on 30 Oct 2026. The IETF Datatracker lists it as an "Active Internet-Draft (individual)" with the standard notice that it "is not endorsed by the IETF and has no formal standing in the IETF standards process." It expires on 3 May 2027 and, like every draft, "may be updated, replaced, or obsoleted by other documents at any time." You use it as the name of a normal robots.txt group. This is the draft's own example:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The draft says this "expresses a preference against AI training for all paths except those under /press-kit/". It sets three rules you need to know. First, "Consumers MUST match policy tokens exactly, apart from case". Second, "wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token." Third, "If no group contains the __usage_ai-training policy token [...] then no preference has been expressed." The token "is not a directive for crawling behavior": the draft calls usage preferences and traffic management "independent and orthogonal", and says an operator who wants to limit crawling "MAY use groups for crawlers' product tokens".

Who reads it, according to the providers' own pages:

  • Microsoft: "Starting February 1, 2027, Microsoft will honor the new preferences as it crawls". The Bing post shows the group User-Agent: __usage_ai-training with Disallow: / for a whole site.
  • Google: its crawler documentation says Google "intends to support the AIPREF shared machine-readable training preference signal (with "user-agent: __usage_ai-training") in early 2027."
  • Anthropic: Kelley wrote on the IETF ai-control list on 30 Oct 2026 that "Anthropic has already started implementing support for this draft", and also: "It'll take us some time to add support across our services".
  • OpenAI: only a press report so far. PPC Land writes that OpenAI aims "to implement the signal in the coming months". OpenAI's site answered our requests with HTTP 403, and its crawler page did not mention the token when we checked on 31 Oct 2026.

So who has actually put the token in a file?

How many top sites use it on day two?

We tested the 500 highest-ranked domains of the Tranco list. Method box. Period: 31 Oct 2026, 19:48 to 19:57 UTC (a first run at 19:20 UTC gave the same token count). Sample: the first 500 rows of Tranco list 746VX (created 30 Oct 2026, 22:00 UTC); the daily download has the same 500 rows. For each domain the script requested https://<domain>/robots.txt once (then www. if that failed), with a 10 second timeout and at most 2 requests per second. A file counts as readable when it answered HTTP 200, was not empty and was not an HTML page. The script reads User-agent lines and the rules of each group, plus Content-Signal lines. Limits: one fetch per domain, no retries, and many of the top names are infrastructure domains (CDNs, DNS, ad and API services) that serve no robots.txt at all. We did not classify them.

Measure (500 domains, 31 Oct 2026) Count
Readable robots.txt 227
Not readable (no answer, error, 404, 403, empty or HTML) 273
Files with a __usage_ai-training User-agent line 2 (x.com, twitter.com: one identical file)
Files with a misspelled variant of the token 0
Files naming Google-Extended 56 (33 with Disallow: / in that group)
Files naming GPTBot 57 (38 with Disallow: /)
Files naming ClaudeBot 56 (38 with Disallow: /)
Files naming at least one of the three 68 of 227
Files with a Content-Signal line 8

The two token files are byte for byte the same, so this is one adopter, not two. Its group is User-agent: __usage_ai-training with Disallow: /. The comment above it says the token is "honored by Microsoft, Google, OpenAI, and Anthropic" and that "User-agent: * never matches it". That is X's description, not the providers': none of them reads the token yet, and the dates above are the ones they give.

Compare that with the vendor groups: 68 of the 227 readable files name GPTBot, ClaudeBot or Google-Extended, and 38 block GPTBot from the whole site. Those 68 files are exactly the ones the shared token is meant to simplify, and on day two none of them had added it. Adoption is near zero, which is what a draft released the previous afternoon should look like. It tells you nothing about what happens once providers read the file. Before you add the group, how does a plain parser handle it?

How does a classic robots.txt parser read the token?

A parser that knows nothing about the draft treats __usage_ai-training as the name of one more crawler. We checked this with Python 3.12.6's standard urllib.robotparser, using a file with a * group and the draft's example group:

from urllib.robotparser import RobotFileParser

robots = """\
User-agent: *
Disallow: /private/

User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
"""
rp = RobotFileParser()
rp.parse(robots.splitlines())

for agent in ("GPTBot", "Googlebot", "__usage_ai-training"):
    for path in ("/", "/private/x", "/press-kit/logo.png"):
        print(f"{agent:20} {path:20} can_fetch={rp.can_fetch(agent, path)}")

GPTBot               /                    can_fetch=True
GPTBot               /private/x           can_fetch=False
GPTBot               /press-kit/logo.png  can_fetch=True
Googlebot            /                    can_fetch=True
Googlebot            /private/x           can_fetch=False
Googlebot            /press-kit/logo.png  can_fetch=True
__usage_ai-training  /                    can_fetch=False
__usage_ai-training  /private/x           can_fetch=False
__usage_ai-training  /press-kit/logo.png  can_fetch=False

Two results matter. First, GPTBot and Googlebot get True for /: the token group's Disallow: / does not touch them, so adding the group cannot block a crawler by accident. Second, the last row is False for /press-kit/logo.png, although the draft's example means that path to stay open. This parser takes the first matching rule in file order, while the draft keeps the "longest-match precedence" of RFC 9309. We swapped the two lines:

Disallow first: False
Allow first:    True

So put the Allow lines before the Disallow line: then first-match and longest-match parsers agree. The second trap is the * rule. With only User-agent: * and Disallow: /private/ in the file, the same library returned False for /private/x with the token as the agent name, because it falls back to *. The draft forbids exactly that, so a provider has to code the exception by hand. Our robots.txt validator applies the draft's rule and shows what your file states for the token. What should the file say meanwhile?

What should you put in robots.txt today?

Keep the vendor lines and add the shared group. A vendor token such as Google-Extended is what the draft calls a "proprietary policy token": a consumer "MUST NOT treat it as expressing a preference in the context of this document", so a Google-Extended block is not a shared opt-out for anyone else. Anthropic writes that the draft "complements our existing crawler controls rather than replacing them", and Google writes that "Existing company-specific controls will continue to function alongside this new joint signal."

Provider Keep this line today Reads __usage_ai-training Source
Google Google-Extended Planned for early 2027. Google plans to read a token opt-out as a Google-Extended disallow when Google-Extended is not explicitly specified Google crawler docs and Search Central Blog, 30 Oct 2026
Microsoft Nothing extra; page-level ai-training tags if you need page control From 1 Feb 2027. NoArchive and NoCache stop counting as a preference after 1 Mar 2027 Bing Blog, 30 Oct 2026
Anthropic ClaudeBot Implementation started, no date IETF ai-control list, 30 Oct 2026
OpenAI GPTBot "the coming months", per PPC Land; no OpenAI page we could read confirms it PPC Land, 31 Oct 2026

A file that opts out of training everywhere and keeps one folder open to it:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: __usage_ai-training
Allow: /press-kit/
Disallow: /

The robots.txt generator writes this group for you, with the Allow lines first: tick the option to opt out of AI training at participating providers and list the folders you want to keep open.

Two readings to watch. Google: its crawler page says Google-Extended covers training and also "grounding [...] in Gemini Apps", its blog says a token opt-out would be read "as if the publisher had disallowed Google-Extended", and it adds that "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." Put together, the shared token alone could also take your pages out of Gemini grounding. If you want training off but grounding on, an explicit Google-Extended group decides for Google, because Google reads the token that way only if "Google-Extended is not explicitly specified". Microsoft: grounding is a separate, page-level setting only ("There is no site-wide robots.txt grounding preference"), so the token changes nothing about Copilot answers. In both cases the token states a preference about use, and as Microsoft writes, "These settings govern use of content, not crawl access."

Takeaway: add the group now, keep every vendor line, and put Allow lines first. Then paste the file into the robots.txt validator to see what it states for the token and how each AI bot reads it.

Common follow-up questions

Does the token replace GPTBot, ClaudeBot and Google-Extended lines?

No. Anthropic says the draft complements its crawler controls, and Google says existing controls keep working alongside it. Keep your vendor groups; see how to block AI crawlers for the full list of tokens.

Will adding the token block Googlebot or Bingbot?

No. The draft says the token is not a crawl directive and that a * group does not match it. In our parser test, Googlebot and GPTBot were unaffected by the token group. Search crawling stays governed by their own rules.

Is the token an official standard?

No. It is an individual Internet-Draft with no IETF endorsement and an expiry date of 3 May 2027. Providers say they will read it, and the draft may still change; robots.txt basics explains the standard it builds on.

What if my file already has User-agent: * rules?

They stay as they are, and they never count as a preference about AI training: the draft says a file without a token group expresses no preference. Only a group that names __usage_ai-training does. For how large sites use vendor groups today, see our study of 151 robots.txt files.

Sources

Checked 31 October 2026.