Why Google isn't crawling your site: causes and how to check each

1. robots.txt disallows the page

If a Disallow rule matches the URL for Googlebot, Google will not fetch it. Run your address through the checker and read the Googlebot row. Remember a group that names Googlebot replaces the * group for it.

2. The server answers with errors

Google warns that an inappropriate response code can affect how your site appears. A 5xx on /robots.txt makes Google stop crawling the whole site for the first 12 hours of the error. Check with curl -I for both /robots.txt and the page.

3. The page says noindex

A noindex meta tag or X-Robots-Tag header keeps the page out of the index. The checker reports both. Google's robots.txt introduction is clear that robots.txt and noindex are different tools.

4. The content only appears after JavaScript

Googlebot can render JavaScript, but a page whose HTML is an empty shell is slower and riskier to process, and many other crawlers do not render at all. Use see a page as a bot to look at the raw HTML.

5. Google does not know the URL yet

New or unlinked pages need to be discovered, through links from other pages or a sitemap. Reference your sitemap in robots.txt with a Sitemap: line and link new pages from existing ones.

Is it really Googlebot?

Google's crawlers identify themselves by user agent, source IP and reverse DNS hostname; see Google's overview of crawlers. A user agent alone proves nothing, so verify an unexpected visitor before you act on it.

Where to look next

Google Search Console shows what Google found, crawled and indexed for a property you own. For the bot side, read the Googlebot page and the other search engine crawlers.