Guides: AI crawlers, robots.txt and llms.txt
Short, sourced guides for site owners. Each states the answer first and links the operators' own documentation.
- How to let AI assistants read your site (and which bots to block)To let AI assistants read your site, allow their search and user-triggered bots in robots.txt, make sure every public page delivers its text as plain HTML without JavaScript, and check that your firewall does not block the same bots. Whether training crawlers may also read your site is a separate choice.Updated
- llms.txt: what it is and what the evidence saysllms.txt is a proposed convention: a Markdown file at /llms.txt that summarizes a site and links to its key pages for language models. Evidence that AI crawlers use it is weak: in one large sample, 97 percent of llms.txt files received no requests at all. It is cheap to add, but do not expect a ranking or citation effect.Updated
- robots.txt explained: syntax, matching rules and where the file goesrobots.txt is a plain text file at the root of a host that tells crawlers which paths they may fetch. A crawler uses only the one group that names it most specifically, and within that group the most specific (longest) matching rule wins. It controls crawling, not indexing.Updated
- robots.txt mistakes, and how to test your fileThe most common robots.txt mistakes are using it to hide pages (it does not), blocking a page you want deindexed so Google never sees the noindex, and a server error on the file itself. Test by fetching the file, checking the status code and evaluating the rules for the bot and path you care about.Updated
- Why Google isn't crawling your site: causes and how to check eachWhen Google is not crawling a site, the cause is usually one of five things: robots.txt disallows the page, the server returns errors, the page says noindex, the content only appears after JavaScript, or Google has not discovered the URL. Check them in that order.Updated
- How to block AI bots in nginx by user agentIn nginx, match the User-Agent header with a `map` block and return 403 for the bots you want to refuse. This stops bots that ignore robots.txt, but it only works against honest user agents: a bot can send any string.Updated
- How to block AI bots in Apache with .htaccessIn Apache, use mod_rewrite to match the User-Agent header and answer named bots with 403 Forbidden. Like all user-agent blocking it only stops bots that send their real name, so keep the matching robots.txt rule too.Updated
- How to block AI crawlers on CloudflareCloudflare's AI Crawl Control lets you allow or block individual AI crawlers and track which ones follow your robots.txt. Cloudflare says it is available on all plans. Use it together with robots.txt, and check that it does not block the search bots you want.Updated
- How to block AI bots on VercelVercel's Firewall has an AI Bots Managed Ruleset, available on all plans, that you can set to Log or Deny. It is off by default (shown as Allow). Deny blocks all traffic Vercel identifies as AI bots, so it also blocks the AI search bots you may want.Updated
Looking for a specific bot? Browse all bots.