llms.txt: what it is and what the evidence says
llms.txt is a proposed convention: a Markdown file at /llms.txt that summarizes a site and links to its key pages for language models. Evidence that AI crawlers use it is weak: in one large sample, 97 percent of llms.txt files received no requests at all. It is cheap to add, but do not expect a ranking or citation effect.
By the Crawlerpeek editorial team · Updated · How we check
What it is
Jeremy Howard proposed llms.txt on 3 September 2024 at llmstxt.org. The file is Markdown with a fixed order: a required H1 with the site name, a blockquote summary, optional details, and H2 sections that hold lists of links. An H2 section named “Optional” marks links an agent can skip.
A minimal template
# Example Shop > Example Shop sells handmade ceramics and ships within the EU. ## Key pages - [Products](https://example.com/products): all products with prices - [Shipping](https://example.com/shipping): delivery times and costs ## Optional - [Blog](https://example.com/blog)
What the data says
- Ahrefs analyzed 137,210 domains with traffic in May 2026. About 28 percent published an llms.txt. 97 percent of those files received zero requests in May 2026.
- In the roughly 3 percent that were requested at all, 19.5 percent of fetches came from named AI tools. Ahrefs notes a fetch does not mean the file was read, so that is an upper bound.
- Google's documentation on AI features says you do not need new machine-readable files, AI text files or markup to appear in them. It does not mention llms.txt by name.
Should you add one?
It takes minutes and does no harm, so adding one is reasonable if you already have documentation or a clear list of key pages. It is not a substitute for what matters more: crawlable pages whose text is in the HTML, and robots.txt rules that allow the bots you want. The checker tells you whether your site has an llms.txt and whether it follows the format.
How we measure our own
This site publishes an llms.txt and counts requests from known AI bots to it. The numbers are on the data page, with a note when the data is still too thin to say anything.
Follow-up questions
Do ChatGPT, Claude or Perplexity read llms.txt?
We found no official statement from OpenAI, Anthropic or Perplexity that their crawlers use it for ranking or citing. Check the official pages linked from each bot page.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers what they may fetch. llms.txt is an optional summary meant to help a model understand a site.
See how your own site does.Run the checker