Which crawlers send Web Bot Auth headers? 34 hours of requests to a small site

Nine named crawlers sent a Signature or Signature-Agent header on 350 of the 1,967 requests that named a crawler, between 8 and 9 November 2026. Here is who, who did not, and what the headers do and do not prove.

By the Crawlerpeek editorial team · Updated · How we check

The short answer

In 34 hours of requests to crawlerpeek.com, nine crawler names sent a Signature or Signature-Agent header on 350 of their requests: ExaSearchBot, AIWebIndex, AhrefsBot, BrickBlueBot, DuckDuckBot, ExaUser, AntiCloneBot, DuckAssistBot and Shap-User. Twenty other named crawlers, among them OAI-SearchBot (978 requests), YandexBot (282), PerplexityBot, Googlebot, bingbot and Amazonbot, sent neither header once. That makes 350 of the 1,967 requests from named crawlers, or 17.8 percent.

**A header is not a verified signature.** We counted whether the headers were present. We did not fetch any operator's key directory and did not check a single signature, so this guide shows who sends the headers, not who passes the check.

What we counted

Window: **8 November 2026, 08:00:00 UTC up to 9 November 2026, 18:00:00 UTC**, 34 hours, with no gap in the server's request log. Host: crawlerpeek.com only. The log is the web server's own JSON request log; the other sites on the same server are not part of any figure here.

  • **53,721 requests** to crawlerpeek.com in the window. Most are people and the site's own monitoring.
  • **1,967 requests** carried a user agent that names one of 29 crawlers. These are the rows of the table and the CSV. A user agent is only a claim, so some of them may not be from the operator.
  • For each crawler we counted the requests with a Signature header, a Signature-Agent header, a Signature-Input header, and with at least one of Signature or Signature-Agent. For Signature-Agent we kept only the host name it points to. No signature value, address or visitor data was kept.

The log only reaches back a few days and has gaps before 8 November, so we used the one stretch without a gap rather than stitching days together.

Crawlers that sent the headers

CrawlerRequestsWith Signature or Signature-AgentShareSignature-Agent points to
ExaSearchBot188188100 %crawler.exa.ai
AIWebIndex114114100 %api.lyrenth.com
AhrefsBot161487.5 %ahrefs.com
BrickBlueBot1414100 %brick.blue
DuckDuckBot99100 %bot.duckduckgo.com
ExaUser77100 %exauser.crawler.exa.ai
AntiCloneBot22100 %anticlone.de
DuckAssistBot11100 %assistbot.duckduckgo.com
Shap-User11100 %shap-user.parallel.ai

In every row, the three headers Signature, Signature-Input and Signature-Agent appeared together; the CSV has each count. The host in the last column is where the request says the operator's public key directory lives. Two operators say the same on their own pages (read 9 November 2026): Exa says every ExaUser request is signed with Web Bot Auth and publishes the key directory at /.well-known/http-message-signatures-directory, and Lyrenth says AIWebIndex requests can carry a signature with its key directory under api.lyrenth.com. Both match what we saw: 7 of 7 ExaUser and 114 of 114 AIWebIndex requests carried the headers. For the other seven crawlers in the table we did not read an operator statement about signing.

AhrefsBot is the only name with a gap: 14 of 16 requests had the headers and 2 did not. We cannot tell from the log why.

Crawlers that sent neither header

CrawlerRequests
OAI-SearchBot978
YandexBot282
PerplexityBot61
DotBot60
meta-webindexer58
Amazonbot46
Googlebot29
ChatGPT-User23
SemrushBot14
Applebot13
MJ12bot12
meta-externalagent10
bingbot7
Googlebot-Image6
KeenableBot6
AdsBot-Google3
facebookexternalhit3
Qwantbot2
GPTBot1
Bravebot1

Keenable says on its bot page that it plans to sign requests but they are not signed yet. That matches the 6 KeenableBot requests we saw, none with the headers. For the others we have no statement either way, so a row of zeros means "no headers in this window", not "the operator does not support signing". Some crawlers may sign only part of their traffic, and some only from certain systems.

Window caveats: bots visit in bursts. ExaSearchBot and AIWebIndex account for 302 of the 350 requests with headers, and one crawler can dominate a day. Read the counts as a snapshot of a small site, not as market share.

The per-crawler counts are public: download the CSV (29 rows, same numbers as the tables above) and reuse it under CC BY 4.0 if you credit Crawlerpeek with a link to this page.

What the headers are

Web Bot Auth lets a crawler sign its requests with a private key and publish the matching public key at a known address. In the requests above, Signature carries the signature, Signature-Input says which parts of the request it covers, and Signature-Agent points to the operator's key directory. Amazon describes the check for its product crawler the same way: fetch the public key from the directory listed in the Signature-Agent header and validate the signature.

How a crawler signs its requests and how a site owner checks the signature is the subject of the explainer on Web Bot Auth. Read more: Web Bot Auth: How to Verify Signed Bot and Agent Requests.

What a signature does not tell you

  • **Presence is not proof.** Anyone can send a Signature header. Only a signature that checks out against the key in the operator's directory shows that the request came from the key's owner.
  • **It says nothing about robots.txt.** A signed crawler can still ignore your rules, and an unsigned one can follow them. Read what each operator says on its own page, as in our bot pages.
  • **It does not replace the other checks.** Where an operator publishes IP ranges or a reverse-DNS method, those still apply (how fake crawlers look in our logs).
  • **It is only as good as the key directory.** If a request names a directory you do not trust, the signature proves nothing about who the operator really is.

How to count it in your own logs

Most servers do not log request headers by default. Caddy's JSON log does, so a count per user agent is one command:

jq -r 'select(.request.headers["Signature-Agent"] != null) | .request.headers["User-Agent"][0]' /var/log/caddy/access.log | sort | uniq -c | sort -rn

For nginx, log whether the header was present instead of its value, so no signature ends up in the file. Put the map in the http block:

map $http_signature_agent $has_sigagent { "" 0; default 1; }
log_format wba '$time_iso8601 $has_sigagent "$http_user_agent"';
access_log /var/log/nginx/wba.log wba;

For Apache:

SetEnvIf Signature-Agent ".+" has_sigagent=1
LogFormat "%{%Y-%m-%dT%H:%M:%S}t %{has_sigagent}e \"%{User-Agent}i\"" wba
CustomLog /var/log/apache2/wba.log wba

In the nginx and Apache files a request without the header shows 0 or -. Count the lines per user agent and compare with the total, as in the table above. Checking the signature itself needs the operator's key directory and a library for HTTP message signatures (RFC 9421); neither command above does that.

Data and limits

Data: the server request log of crawlerpeek.com, 8 November 2026 08:00:00 UTC to 9 November 2026 18:00:00 UTC. Operator statements from their own pages, read on 9 November 2026. One small site over 34 hours is a sample, and the numbers will differ on yours.

Follow-up questions

Which crawlers send Web Bot Auth headers?

In 34 hours of requests to crawlerpeek.com (8 to 9 November 2026), ExaSearchBot, AIWebIndex, AhrefsBot, BrickBlueBot, DuckDuckBot, ExaUser, AntiCloneBot, DuckAssistBot and Shap-User sent a Signature or Signature-Agent header. OAI-SearchBot, YandexBot, PerplexityBot, Googlebot and bingbot sent neither. These counts show header presence, not verified signatures.

Does a Signature header prove a request is from the crawler it names?

No. Anyone can send the header. A request is verified only when its signature checks out against the public key in the operator's key directory, which the Signature-Agent header points to.

How do I see which bots send Signature-Agent in my own logs?

Log the header, or whether it was present, next to the user agent and count by user agent. The commands for Caddy, nginx and Apache are in the guide above. In nginx and Apache, log presence rather than the value.

Does a signed crawler obey robots.txt?

The signature does not say. It shows who sent the request, not what the sender does with your rules. Each operator documents its robots.txt behavior separately.

See how your own site does.Run the checker

Keep reading