robots.txt mistakes, and how to test your file
The most common robots.txt mistakes are using it to hide pages (it does not), blocking a page you want deindexed so Google never sees the noindex, and a server error on the file itself. Test by fetching the file, checking the status code and evaluating the rules for the bot and path you care about.
By the Crawlerpeek editorial team · Updated · How we check
Mistakes that matter
| Mistake | What actually happens | What to do instead |
|---|---|---|
| Using Disallow to keep a page out of Google | Google says robots.txt is not a mechanism for keeping a page out of Google. A blocked URL can still be indexed if other sites link to it, without a description. | Use noindex or password protection. |
| Blocking a page that carries noindex | The crawler cannot fetch the page, so it never sees the noindex. | Allow crawling until the page has dropped out. |
| Putting robots.txt in the wrong place | Crawlers only look at the root of the exact host; a file in a subdirectory or on another subdomain is ignored. | Serve /robots.txt on every host. |
| A 5xx error on /robots.txt | Google stops crawling the whole site for the first 12 hours of the error and later falls back to a cached copy. | Make sure the file always returns 200, or 404 if you have none. |
Expecting a specific group to add to * | A crawler uses only its most specific group, so rules in User-agent: * do not apply to a bot that has its own group. | Repeat shared rules inside each bot's group. |
| Wrong case in paths | Path values are case-sensitive. | Match the real URL case. |
| Relying on it for security or against bad bots | Compliance is voluntary and some bots ignore it. | Use server or firewall rules; see the blocking guides. |
Sources: Google on what robots.txt is for and on the file specification.
How to test
- Fetch the file and read the status:
curl -i https://example.com/robots.txtshould show 200 and plain text, not an HTML page. - Evaluate the rules for the bot and the path that matter. The checker does this for every bot in the directory using the longest-match rule of RFC 9309.
- Look at the page as the bot receives it with see a page as a bot; robots.txt can allow a bot that then gets an empty HTML shell.
- After changing the file, wait: operators cache robots.txt for hours (Google up to 24 hours).
See how your own site does.Run the checker