AIWebIndex (Lyrenth): what it is and how to allow or block it
Updated · checked against the operator's documentation on
AIWebIndex is the crawler behind Lyrenth (Aleksma Ai, Inc.), an AI-readable web index that serves public pages as attributed documents. Lyrenth says it indexes content and does not train foundation models on it, follows RFC 9309 and fails closed if robots.txt cannot be read. Block it with User-agent: AIWebIndex. Its robots.txt token is AIWebIndex.
Seen AIWebIndex/2.0 in your logs?
AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)The example string from Lyrenth's documentation. Version numbers change; the token AIWebIndex stays.
- What it is
- An AI search crawler run by Lyrenth.
- Honors robots.txt
- YesYes, per Lyrenth.
- Real or fake?
- Can be verifiedThe user agent is only a claim. Check the requesting IP against Lyrenth's published IP ranges.
- If you block it
- Your pages are no longer fetched for the Lyrenth index.
Facts
- Operator
- Lyrenth
- Category
- AI search crawler
- robots.txt token
AIWebIndex
- Honors robots.txt?
- Yes, per Lyrenth. Lyrenth says it follows RFC 9309, honors Disallow, Crawl-delay, Sitemap and HTTP 429/503 backoff, and fails closed if robots.txt cannot be read. Its opt-out is User-agent: AIWebIndex with Disallow: /. It says it fetches at most once every 2 seconds per domain, or the site's Crawl-delay if longer; owner-requested rechecks skip that wait but robots.txt still applies.
- User-agent example
AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)
- Official source
- Lyrenth: AIWebIndex
- Published IP ranges
- https://lyrenth.com/bot/ip-ranges.json
- Compared with the source
What it is used for
Per Lyrenth, AIWebIndex is the crawler behind an AI-readable web index that serves public pages as attributed AIDocuments. It says it indexes content and does not train foundation models on it. It also names AIWebIndex-Agent for on-demand fetches requested by a customer, and a verification probe string used to check domain ownership.
Builds the index an AI assistant searches when it answers. Blocking it can keep you out of that assistant's answers.
What happens if you block it
Your pages are no longer fetched for the Lyrenth index. Nothing about your Google rankings changes.
Lyrenth publishes an IP list at https://lyrenth.com/bot/ip-ranges.json, says every crawler IP has forward-confirmed reverse DNS under lyrenth.com, and says requests can carry a Web Bot Auth signature with the key directory at api.lyrenth.com/.well-known/http-message-signatures-directory. It also says Cloudflare lists AIWebIndex as a verified search bot; we did not check that.
Is it really AIWebIndex?
Anyone can send AIWebIndex as a user agent, so a log line proves nothing by itself. The way to tell is the requesting IP address.
- Compare the IP address with the Lyrenth: published IP ranges. A request from outside those ranges is not the operator's crawler.
- Guide: robots.txt syntax: groups, wildcards and matching rules.
- Guide: Which crawlers send Web Bot Auth headers? 34 hours of requests to a small site.
- Paste any user agent from your logs into the user-agent lookup, or follow the scripts in Fake Googlebots and PHP scanners.
How to allow or block AIWebIndex
Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.
User-agent: AIWebIndex Allow: /
User-agent: AIWebIndex Disallow: /
Block AIWebIndex at the server
robots.txt asks; a server rule refuses. These answer 403 to any request whose user agent contains aiwebindex (case-insensitive). They stop only bots that name themselves honestly, so keep the robots.txt group as well.
# server { ... }
if ($http_user_agent ~* "AIWebIndex") {
return 403;
}RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} AIWebIndex [NC]
RewriteRule .* - [F,L]Step by step, with a test command: Block AI bots in nginx by user agent · Block AI bots in Apache with .htaccess · Block AI crawlers on Cloudflare · Block AI bots on Vercel
Does your site allow AIWebIndex today? The checker evaluates your robots.txt for this and every other bot.Check my site
What we see on this site
No request identifying as AIWebIndex has been recorded in the last 30 days, so there is nothing to show yet. We never fill this in with estimates.
Counts come from this site's own server; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.
Questions about AIWebIndex
What is AIWebIndex?
The crawler of Lyrenth, an AI-readable web index run by Aleksma Ai, Inc. Per Lyrenth, AIWebIndex is the crawler behind an AI-readable web index that serves public pages as attributed AIDocuments. It says it indexes content and does not train foundation models on it. It also names AIWebIndex-Agent for on-demand fetches requested by a customer, and a verification probe string used to check domain ownership.
Is AIWebIndex safe and legitimate?
The real AIWebIndex is an AI search crawler run by Lyrenth. A user-agent string is only a claim, though, so anyone can send it. Compare the requesting IP address with the published IP ranges.
Does AIWebIndex obey robots.txt?
Yes, per Lyrenth. Lyrenth says it follows RFC 9309, honors Disallow, Crawl-delay, Sitemap and HTTP 429/503 backoff, and fails closed if robots.txt cannot be read. Its opt-out is User-agent: AIWebIndex with Disallow: /. It says it fetches at most once every 2 seconds per domain, or the site's Crawl-delay if longer; owner-requested rechecks skip that wait but robots.txt still applies.
How do I block AIWebIndex?
Add a group to the robots.txt at the root of your domain: User-agent: AIWebIndex followed by Disallow: /. A group that names the bot overrides a general User-agent: * group.
Will blocking AIWebIndex hurt my Google rankings or AI visibility?
It can. This bot builds the index an AI assistant searches when it answers, so blocking it can keep you out of that assistant's answers. Your pages are no longer fetched for the Lyrenth index. Nothing about your Google rankings changes.
What does "AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)" mean in my logs?
It is the user-agent string in Lyrenth's documentation for AIWebIndex, an AI search crawler run by Lyrenth. The part that robots.txt rules match is the token AIWebIndex. The string is only a claim, because anyone can send it; the section "Is it really AIWebIndex?" above shows how to check a request.
Related bots
Other AI search crawlers
- OAI-SearchBot OpenAIOpenAI's crawler that surfaces websites in ChatGPT's search features.
- Claude-SearchBot AnthropicAnthropic's crawler that analyzes online content to improve the quality of Claude's search results.
- PerplexityBot PerplexityPerplexity's crawler that surfaces and links websites in Perplexity search results.
- ExaSearchBot ExaExa's crawler that discovers and indexes public pages for its search and retrieval service.
All AI Search Crawlers · Bot directory · Which bots to allow or block · robots.txt generator · See a page as a bot