robots.txt generator for AI bots and search crawlers
Pick what each group of bots may do and copy the result into the robots.txt at the root of your domain. Every token comes from the operator's own documentation; you can look each one up in the bot directory.
Your robots.txt
User-agent: *
Allow: /
"No rule" leaves a bot to the general group at the end. A group naming a bot overrides it.
Before you publish
The file must be served at /robots.txt on the exact host it should govern; each subdomain needs its own.
robots.txt is a request. It does not hide pages from search: use noindex for that. Read common robots.txt mistakes.
Slackbot does not honor robots.txt and some user-triggered fetchers may bypass it; see their pages for details.
Afterwards, run the checker on your site to confirm the result.
