Crawlerpeek
Check a site

Web Bot Auth: How to Verify Signed Bot and Agent Requests

Web Bot Auth lets a bot sign its HTTP requests with a key you can fetch. We read five key directories on 1 November 2026 and ran a real RFC 9421 test.

Published
12 min read
  • web bot auth
  • http message signatures
  • bot verification
  • ai agents
  • signature-agent
  • key directory

Web Bot Auth is an IETF draft protocol in which a bot signs each HTTP request with a private key and names a public key directory, so a server can verify the sender without an IP list. On this site, 619 of the 1,570 bot requests our IP check could not verify came from two bots that say they sign.

As of 1 November 2026.

What does a signed bot request look like?

A Web Bot Auth request carries three extra headers. Signature holds the signature bytes, Signature-Input lists what was signed and the parameters, and Signature-Agent names the HTTPS address of the sender's key directory. If a request carries other signatures too, the draft lets a server discard those whose tag is not web-bot-auth.

This is how such a request arrives. The headers come from a test vector in the IETF draft, with an invented host and agent; the nonce and signature are shortened here and printed in full in the script below.

GET /guides HTTP/1.1
Host: example.com
Signature-Agent: sig2="https://signature-agent.test"
Signature-Input: sig2=("@authority" "signature-agent";key="sig2");created=1735689600;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";alg="ed25519";expires=4889289600;nonce="n9p433xm+NJ3...";tag="web-bot-auth"
Signature: sig2=:CGokAoY/FrPpyuzTe1F...:

Each field has a job, and the draft sets rules for most of them:

Field What it carries Rule in the draft
Signature-Agent HTTPS URL of the sender's key directory A signed request must carry it; signers must use the dictionary form shown above
keyid Which key signed Base64url JWK SHA-256 thumbprint of the key
created, expires Signature lifetime expires recommended no more than 24 hours after created
tag Marks a Web Bot Auth signature Must be web-bot-auth

The protocol is not a standard yet. The IETF datatracker lists draft-ietf-webbotauth-httpsig-protocol-01, dated 29 October 2026, as an Active Internet-Draft of the Web Bot Auth working group, with the intended status Standards Track and an expiry date of 2 May 2027. It descends from two earlier individual drafts, architecture and directory, both last updated on 2 March 2026 and now marked as replaced. The signing mechanism underneath, RFC 9421, is a Standards Track RFC from February 2024.

So which operators publish keys you can fetch?

Which bot operators publish a Web Bot Auth key directory?

Five operators served a key directory at /.well-known/http-message-signatures-directory when we fetched them on 1 November 2026: Google-Agent, Exa, Parallel's ShapBot, FossickBot and chatgpt.com. All five answered HTTP 200 with the media type application/http-message-signatures-directory+json, and every key was Ed25519.

Method box. Period: 1 November 2026, at about 09:39 UTC. We fetched each directory with curl, one request per second, and saved every raw response. A saved script (directories.py) counts the keys and builds the table. The last column records whether the operator's own page names an IP list; where it does, we fetched that file too. Limits: five operators we picked, not a census. OpenAI's help article on ChatGPT signing refused our automated fetch with HTTP 403, so the chatgpt.com row shows only what the directory serves, not what OpenAI documents.

Operator (directory host) Keys kid equals thumbprint Cache-Control Also publishes an IP list
Google-Agent (agent.bot.goog) 5 0 of 5 public, max-age=3600 Yes, user-triggered-agents.json, 20 prefixes
Exa (crawler.exa.ai) 1 1 of 1 max-age=300 None named on its page
ShapBot, Parallel (shapbot.parallel.ai) 2 2 of 2 max-age=300 Yes, shapbot.json, 10 prefixes
FossickBot (fossick.bot) 1 1 of 1 max-age=86400 None named on its page
chatgpt.com 1 1 of 1 no-store, no-transform Not determined

Each bot has a profile with its robots.txt facts: Google-Agent, ExaSearchBot, ShapBot and FossickBot. Three details matter if you write a verifier. Every directory response carried its own Signature and Signature-Input headers. Google's five kid values are six-character labels such as DYiMjA, while the draft says a kid in a well-known directory must equal the key's thumbprint, so select keys by keyid against both the kid and a thumbprint you compute. And the chatgpt.com directory adds two members outside the key set, signature_agent set to https://chatgpt.com and purpose set to ai.

Two of the five also publish IP lists. For Exa and FossickBot the signature is the only verification method their pages name, and Exa's page spells out what that means: "Because verification is cryptographic, it holds regardless of the request's IP address." Google's position differs. Its experimental guide says "We don't sign every request of a particular agent" and tells you to fall back to IP-based verification.

You know where the keys are. How do you check a signature against one?

How do you verify a Web Bot Auth signature yourself?

Rebuild the signature base from the request, find the public key named by keyid in the sender's directory, and verify the Ed25519 signature over that base. The script below does this for the draft's test vector and checks that the keyid equals the key's thumbprint.

The steps follow the draft:

  1. Read Signature-Input. Drop signatures whose tag is not web-bot-auth or whose expires has passed.
  2. Fetch the directory from the Signature-Agent URL over HTTPS. The draft says to treat any status other than 200 as a failure and not to follow redirects. Cache by Cache-Control.
  3. Pick the key whose kid or thumbprint matches keyid. The draft says lookups must use the pair of URL and key, never the key alone.
  4. Write one line per covered component, then the @signature-params line. That text is the signature base.
  5. Verify the signature over the base with the public key.

Method box (tested command). Python 3.12.6 with cryptography 45.0.5, run on 1 November 2026. The test vector is Appendix E.2.1 of the draft, which uses the Ed25519 test key from RFC 9421, Appendix B.1.4. Tests 2 to 4 change one thing each and must fail. Limits: the script handles only the @authority and signature-agent components that the vector uses, rejects others, and does not fetch directories.

import base64, hashlib, json, re, time
from cryptography.exceptions import InvalidSignature
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey

def b64u(s): return base64.urlsafe_b64decode(s + "=" * (-len(s) % 4))

def thumbprint(jwk):  # RFC 7638 over crv, kty, x
    c = json.dumps({"crv": jwk["crv"], "kty": jwk["kty"], "x": jwk["x"]}, separators=(",", ":"))
    return base64.urlsafe_b64encode(hashlib.sha256(c.encode()).digest()).rstrip(b"=").decode()

def verify(h, authority, jwks, now):
    label, rest = h["Signature-Input"].split("=", 1)
    comps, params = re.fullmatch(r"\(([^)]*)\)(;.*)", rest).groups()
    p = {k: v.strip('"') for k, v in re.findall(r';([a-z]+)=("[^"]*"|\d+)', params)}
    if not p.get("tag") == "web-bot-auth": return "no: tag is not web-bot-auth"
    if int(p["expires"]) < now: return "no: signature expired"
    lines = []
    for c in re.findall(r'"[^"]+"(?:;key="[^"]*")?', comps):
        if c.startswith('"@authority"'):
            lines.append(f"{c}: {authority}")
        elif c.startswith('"signature-agent"'):  # dictionary form sig2="https://..." or bare string
            key = re.search(r';key="([^"]*)"', c)
            m = re.fullmatch(re.escape(key.group(1)) + r"=(.*)", h["Signature-Agent"]) if key else None
            lines.append(f"{c}: {m.group(1) if m else h['Signature-Agent']}")
        else:
            return f"no: component {c} not supported here"
    lines.append(f'"@signature-params": ({comps}){params}')
    jwk = next((k for k in jwks["keys"] if p["keyid"] in (k.get("kid"), thumbprint(k))), None)
    if jwk is None: return "no: keyid not in directory"
    sig = base64.b64decode(re.search(re.escape(label) + r"=:([^:]+):", h["Signature"]).group(1))
    try:
        Ed25519PublicKey.from_public_bytes(b64u(jwk["x"])).verify(sig, "\n".join(lines).encode())
    except InvalidSignature:
        return "no: signature does not match"
    return "yes: valid, signed by key " + p["keyid"]

# Test vector: draft-ietf-webbotauth-httpsig-protocol-01, Appendix E.2.1 (RFC 9421 key B.1.4)
H = {"Signature-Agent": 'sig2="https://signature-agent.test"',
     "Signature-Input": 'sig2=("@authority" "signature-agent";key="sig2");created=1735689600;'
         'keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";alg="ed25519";expires=4889289600;'
         'nonce="n9p433xm+NJ3ph3upfBIGmsuwHw387YV7Q/F+6BSpGCVjYCqQw6rznNA8PVVLySrAWsv0hQtFioQb6E1YsauiA==";'
         'tag="web-bot-auth"',
     "Signature": "sig2=:CGokAoY/FrPpyuzTe1Fb2kLTAH9Idw6qSFd1+D77r1firsMRCxeuLu5e6O+ek3zQgDMAzjiXtLVfqE00xnABCQ==:"}
KEYS = {"keys": [{"kty": "OKP", "crv": "Ed25519", "x": "JrQLj5P_89iXES9-vFgrIy29clF9CC_oPPsw3c5D0bs"}]}
OTHER = {"keys": [{"kty": "OKP", "crv": "Ed25519", "x": "f1UrAERwBTuaxqbngzcny0_rm7JycUzgbb0bUUvh0yw"}]}
now = int(time.time())
print("thumbprint of directory key:", thumbprint(KEYS["keys"][0]))
print("1 vector as published      ->", verify(H, "example.com", KEYS, now))
print("2 request for example.org  ->", verify(H, "example.org", KEYS, now))
print("3 Signature-Agent swapped  ->", verify({**H, "Signature-Agent": 'sig2="https://other.test"'}, "example.com", KEYS, now))
print("4 another operator's key   ->", verify(H, "example.com", OTHER, now))

Save it as verify_wba.py, then run it:

$ pip install cryptography
$ python3 -I verify_wba.py
thumbprint of directory key: poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U
1 vector as published      -> yes: valid, signed by key poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U
2 request for example.org  -> no: signature does not match
3 Signature-Agent swapped  -> no: signature does not match
4 another operator's key   -> no: keyid not in directory

Test 1 passes, and the computed thumbprint equals the keyid, as the draft requires. Test 2 fails because @authority is part of the signed text. Test 3 fails because the Signature-Agent value is covered too, and test 4 fails because the key is not in that directory. The @authority part protects against moving the signature to another host, but not against reuse on the same host: the draft says a signature covering @authority alone "verifies against any method, path, or body sent to that authority until it expires". A short expires bounds how long such a reuse can work.

That is the check on paper. What does it look like on a real site?

How many signing bots reached this site, and could our logs check them?

Between 19 and 31 October 2026, ExaSearchBot made 584 requests and FossickBot made 35 on this site, and our IP check marked all 619 as unverified, because neither operator names an IP list. That is 39 percent of the 1,570 requests in the unverified group.

Method box. Period: 19 to 31 October 2026, from our own server logs. Source: the per-URL read API for this site, 10,286 requests in 2,713 rows of path and bot, not truncated. The verified field is true when the request IP sits inside the operator's published list, false when outside, and null when we hold no list for that operator. We summed requests by bot and by verified. Limits: our logs do not record the Signature, Signature-Input or Signature-Agent headers, so we could not check a single signature. The aggregate endpoint for the same period reports 472 requests for ExaSearchBot, 8,236 requests in total and no FossickBot rows; we do not know why the two endpoints differ, and either way no Exa request was verified.

Bot (operator) Requests IP check true IP check null Signature directory
ExaSearchBot (Exa) 584 0 584 Yes
FossickBot 35 0 35 Yes
ShapBot (Parallel) 69 69 0 Yes
Google-Agent 0 0 0 Yes

ShapBot shows what a double check gives you: all 69 requests passed the IP list, so a signature would only confirm it. Google-Agent made no requests that we logged. The other 951 unverified requests came from bots with no key directory in our table, such as MJ12bot and YandexBot, so signatures would not help there. For requests that claim a crawler name and sign nothing, see fake crawlers in server logs.

We cannot say whether any of the 619 requests carried a valid signature. To find out, a site needs to log the three signature headers (or the verification result from its CDN), the Host header and the request time, then verify at the edge or from the log with a script like the one above.

If your CDN verifies for you, what does it check?

What do Cloudflare and AWS WAF verify for you?

Cloudflare and AWS WAF both document Web Bot Auth checks at the edge. Cloudflare's developer docs (last updated 8 October 2026) describe the older string form of Signature-Agent and say the dictionary form fails. AWS WAF Bot Control (page read 1 November 2026) adds a web_bot_auth label with a verification status.

Cloudflare's page says its implementation supports Ed25519 keys and lists, among the reasons a message fails verification, "The message uses the dictionary form from later drafts". The IETF draft requires signers to send the dictionary form, and Google's guide shows the header as g="https://agent.bot.goog". The pages describe different draft revisions, so check which form your provider expects before you write a custom rule. The script above accepts both.

AWS documents four label statuses. web_bot_auth:verified means "Signature successfully validated against public key directory", and web_bot_auth:unknown_bot means "Key ID not found in the key directory". The other two are invalid and expired.

A pass at the edge tells you the signature fit a registered key. Is that enough to let a bot through?

Is a valid signature enough, and what about unsigned requests?

A valid signature proves that a holder of a key published at the sender's URL signed the message. The draft states that it "says nothing about who operates the Agent, whether the Agent is benign, or whether the request is authorized." Unsigned requests stay your problem: a client may choose not to sign.

Signatures add a third check beside the two you already have:

Check What a pass shows What it needs from the operator Covers unsigned requests
IP list Source address is inside a range the operator published A published JSON list; Google and Parallel publish one, Exa and FossickBot name none on their pages Yes
Reverse and forward DNS Address resolves to the operator's domain and back DNS names the operator documents; Google names googlebot.com, google.com and googleusercontent.com Yes
Signature A holder of a published key signed this message A key directory Only requests the operator signs

Keep the IP check for everything unsigned (the GPTBot IP ranges article shows the list-matching method), and treat a valid signature as an extra yes, not as the only gate. To test one address against an operator's published ranges, paste it into the IP checker, which does not check signatures.

Common follow-up questions

What is Web Bot Auth?

Web Bot Auth is an IETF draft protocol, built on RFC 9421 HTTP message signatures, that lets a bot sign its requests and point to a public key directory. A server fetches the key and checks the signature instead of relying on the user agent or an IP list alone.

Is Web Bot Auth an official standard?

Not yet. The IETF datatracker lists draft-ietf-webbotauth-httpsig-protocol-01 of 29 October 2026 as an Active Internet-Draft of the Web Bot Auth working group, with Standards Track as the intended status. Drafts can change before publication. RFC 9421, the signing mechanism it builds on, is a published RFC.

What does the Signature-Agent header contain?

It holds the HTTPS URL of the sender's key directory. The draft's current form is a dictionary such as sig2="https://signature-agent.test", and Google's guide shows g="https://agent.bot.goog". The header names where candidate keys live; the keyid in Signature-Input selects the key.

What does ChatGPT send in Signature-Agent?

We could not confirm it from OpenAI's own page, which refused our automated fetch with HTTP 403. What we can state is that chatgpt.com served a key directory on 1 November 2026 whose body names signature_agent as https://chatgpt.com. Read OpenAI's help article before you write a rule.

Does Web Bot Auth replace IP allowlists?

Not for Google. Its guide says the Web Bot Auth implementation is experimental, that not all Google user agents use it, and that you should keep relying on IP addresses, reverse DNS and user-agent strings while signed traffic rolls out. Exa and FossickBot name no IP list, so for them the signature is the main check.

Sources

Checked 1 November 2026.