How to let AI assistants read your site (and which bots to block)
To let AI assistants read your site, allow their search and user-triggered bots in robots.txt, make sure every public page delivers its text as plain HTML without JavaScript, and check that your firewall does not block the same bots. Whether training crawlers may also read your site is a separate choice.
By the Crawlerpeek editorial team · Updated · How we check
Three kinds of AI bots
Most AI companies run more than one bot, and each has its own robots.txt token. Treating them as one is the most common mistake.
| Type | What it does | Examples | Blocking it means |
|---|---|---|---|
| Search crawler | Builds the index an assistant searches when it answers | OAI-SearchBot, Claude-SearchBot, PerplexityBot | You can drop out of that assistant's answers |
| User-triggered fetcher | Opens one page because a person asked | ChatGPT-User, Claude-User, Perplexity-User | Shared links may not be readable; some operators ignore robots.txt for these |
| Training crawler | Collects pages that may be used to train models | GPTBot, ClaudeBot, CCBot, meta-externalagent | A statement about training, not about being found |
| Control token | Not seen in logs; tells the operator how collected content may be used | Google-Extended, Applebot-Extended | No effect on Google Search or Apple search |
Step 1: check what robots.txt says today
Run your address through the checker. It evaluates your robots.txt for every bot in the table above and shows which ones may fetch the page you entered. A missing robots.txt means everything is allowed.
Step 2: allow the bots you want
A bot follows the most specific group that names it. A group with the bot's own token therefore overrides a general User-agent: * rule. This example allows AI search and user-triggered fetchers:
User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: /
Step 3: decide about training
Allowing or blocking training crawlers is your decision. Per the operators' own documentation it does not change whether you appear in their search features. To opt out of training only:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: meta-externalagent Disallow: /
Step 4: make sure the text is in the HTML
Allowing a bot is useless if the page it receives is empty. Many AI crawlers do not run JavaScript, and sites built as client-side single-page apps (React with Vite, for example) often return an almost empty HTML shell. Some builders work around this: Lovable's FAQ says older apps render in the browser and that it serves pre-rendered copies to verified crawlers, while newer apps use server rendering. Your own host may differ, so look at what your page really returns. The checker shows the title, headline, text length and link count a bot gets without JavaScript. If the main text is missing, prerender the public pages or render them on the server.
Step 5: check your firewall and CDN
robots.txt is a request, not a lock. A CDN or web application firewall can still block bots that robots.txt allows, and some bot-protection features are on by default. If robots.txt allows a bot but it never shows up in your logs, look at your bot-management settings. Operators publish IP ranges so you can allow genuine traffic and refuse forged user agents; see the official page linked on each bot page.
What robots.txt cannot do
- It does not remove content that was already collected.
- It is voluntary: well-behaved bots follow it, others may not.
- Some user-triggered fetchers may ignore it. Operators document this for ChatGPT-User and Perplexity-User.
- A user-agent string is only a claim. Anyone can send
GPTBotas a user agent. Verify with the operator's IP list.
Follow-up questions
Does blocking GPTBot remove my site from ChatGPT?
Not by itself. OpenAI documents GPTBot as the training crawler. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages when a person asks. Each has its own robots.txt token.
Does blocking Google-Extended affect Google Search?
No. Google states that Google-Extended does not impact a site's inclusion in Google Search. It manages use of content for Gemini training and grounding.
Can I trust the user agent in my logs?
Only as a claim. Any client can send any user-agent string. Compare the requesting IP address with the operator's published IP list where one exists.
See how your own site does.Run the checker