AIWebIndex (Lyrenth): what it is and how to allow or block it

Updated · checked against the operator's documentation on

AIWebIndex is the crawler behind Lyrenth (Aleksma Ai, Inc.), an AI-readable web index that serves public pages as attributed documents. Lyrenth says it indexes content and does not train foundation models on it, follows RFC 9309 and fails closed if robots.txt cannot be read. Block it with User-agent: AIWebIndex. Its robots.txt token is AIWebIndex.

Seen AIWebIndex/2.0 in your logs?

AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)

The example string from Lyrenth's documentation. Version numbers change; the token AIWebIndex stays.

What it is
An AI search crawler run by Lyrenth.
Honors robots.txt
YesYes, per Lyrenth.
Real or fake?
Can be verifiedThe user agent is only a claim. Check the requesting IP against Lyrenth's published IP ranges.
If you block it
Your pages are no longer fetched for the Lyrenth index.

Facts

Operator
Lyrenth
Category
AI search crawler
robots.txt token
AIWebIndex
Honors robots.txt?
Yes, per Lyrenth. Lyrenth says it follows RFC 9309, honors Disallow, Crawl-delay, Sitemap and HTTP 429/503 backoff, and fails closed if robots.txt cannot be read. Its opt-out is User-agent: AIWebIndex with Disallow: /. It says it fetches at most once every 2 seconds per domain, or the site's Crawl-delay if longer; owner-requested rechecks skip that wait but robots.txt still applies.
User-agent example
AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)
Official source
Lyrenth: AIWebIndex
Published IP ranges
https://lyrenth.com/bot/ip-ranges.json
Compared with the source

What it is used for

Per Lyrenth, AIWebIndex is the crawler behind an AI-readable web index that serves public pages as attributed AIDocuments. It says it indexes content and does not train foundation models on it. It also names AIWebIndex-Agent for on-demand fetches requested by a customer, and a verification probe string used to check domain ownership.

Builds the index an AI assistant searches when it answers. Blocking it can keep you out of that assistant's answers.

What happens if you block it

Your pages are no longer fetched for the Lyrenth index. Nothing about your Google rankings changes.

Lyrenth publishes an IP list at https://lyrenth.com/bot/ip-ranges.json, says every crawler IP has forward-confirmed reverse DNS under lyrenth.com, and says requests can carry a Web Bot Auth signature with the key directory at api.lyrenth.com/.well-known/http-message-signatures-directory. It also says Cloudflare lists AIWebIndex as a verified search bot; we did not check that.

Is it really AIWebIndex?

Anyone can send AIWebIndex as a user agent, so a log line proves nothing by itself. The way to tell is the requesting IP address.

How to allow or block AIWebIndex

Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.

Allow AIWebIndex
User-agent: AIWebIndex
Allow: /
Block AIWebIndex
User-agent: AIWebIndex
Disallow: /

Block AIWebIndex at the server

robots.txt asks; a server rule refuses. These answer 403 to any request whose user agent contains aiwebindex (case-insensitive). They stop only bots that name themselves honestly, so keep the robots.txt group as well.

nginx
# server { ... }
if ($http_user_agent ~* "AIWebIndex") {
    return 403;
}
Apache (.htaccess)
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} AIWebIndex [NC]
RewriteRule .* - [F,L]

Step by step, with a test command: Block AI bots in nginx by user agent · Block AI bots in Apache with .htaccess · Block AI crawlers on Cloudflare · Block AI bots on Vercel

Does your site allow AIWebIndex today? The checker evaluates your robots.txt for this and every other bot.Check my site

What we see on this site

No request identifying as AIWebIndex has been recorded in the last 30 days, so there is nothing to show yet. We never fill this in with estimates.

Counts come from this site's own server; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.

Questions about AIWebIndex

What is AIWebIndex?

The crawler of Lyrenth, an AI-readable web index run by Aleksma Ai, Inc. Per Lyrenth, AIWebIndex is the crawler behind an AI-readable web index that serves public pages as attributed AIDocuments. It says it indexes content and does not train foundation models on it. It also names AIWebIndex-Agent for on-demand fetches requested by a customer, and a verification probe string used to check domain ownership.

Is AIWebIndex safe and legitimate?

The real AIWebIndex is an AI search crawler run by Lyrenth. A user-agent string is only a claim, though, so anyone can send it. Compare the requesting IP address with the published IP ranges.

Does AIWebIndex obey robots.txt?

Yes, per Lyrenth. Lyrenth says it follows RFC 9309, honors Disallow, Crawl-delay, Sitemap and HTTP 429/503 backoff, and fails closed if robots.txt cannot be read. Its opt-out is User-agent: AIWebIndex with Disallow: /. It says it fetches at most once every 2 seconds per domain, or the site's Crawl-delay if longer; owner-requested rechecks skip that wait but robots.txt still applies.

How do I block AIWebIndex?

Add a group to the robots.txt at the root of your domain: User-agent: AIWebIndex followed by Disallow: /. A group that names the bot overrides a general User-agent: * group.

Will blocking AIWebIndex hurt my Google rankings or AI visibility?

It can. This bot builds the index an AI assistant searches when it answers, so blocking it can keep you out of that assistant's answers. Your pages are no longer fetched for the Lyrenth index. Nothing about your Google rankings changes.

What does "AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)" mean in my logs?

It is the user-agent string in Lyrenth's documentation for AIWebIndex, an AI search crawler run by Lyrenth. The part that robots.txt rules match is the token AIWebIndex. The string is only a claim, because anyone can send it; the section "Is it really AIWebIndex?" above shows how to check a request.

Other AI search crawlers

All AI Search Crawlers · Bot directory · Which bots to allow or block · robots.txt generator · See a page as a bot