Crawlerpeek
Check a site

Server error (5xx) in Search Console: what Google does next and how to find the cause

Your server answered Googlebot with a 500-level status, so the page was not indexed. Find the failing requests in your logs, fix the cause, then validate.

The message in Search Console

Server error (5xx)

Page indexing report, group "Not indexed". Spelling as in Google's help page for the report.

By the Crawlerpeek editorial team · Sources checked · Updated · Editorial standards

What Google says

Your server returned a 500-level error when the page was requested.

Source: Google Search Console Help: Page indexing report, read 2026-10-30.

Common causes

  • The application fails on some URLs: an unhandled exception, a database timeout, a missing dependency.
  • The server is overloaded during crawl peaks or traffic spikes, and a proxy answers 502, 503 or 504.
  • Maintenance mode answers 503 for longer than planned.
  • The robots.txt URL itself fails. Google says that for a 5xx on robots.txt it stops crawling the site for the first 12 hours, then uses the last good version for up to 30 days (Google's robots.txt specification).

How to check

  1. Enter the URL in the AI crawler checker: it shows the HTTP status our request got and whether robots.txt could be fetched. The checker requests the page with a browser user agent and robots.txt with Crawlerpeek's own, never as Googlebot, so a server that treats Googlebot differently can still answer us with 200.
  2. In your access log, filter requests whose user agent says Googlebot and whose status is 5xx, and confirm the IP addresses with verify a crawler IP. Forged Googlebot requests do happen: on our own new web server, 77 requests in 48 hours named Googlebot from addresses outside Google's list (what hit our new server in 48 hours).
  3. Open the Crawl Stats report for Googlebot's crawling history, and run URL Inspection's live test on an affected URL.
  4. Fix the cause, then click Validate fix in the Page indexing report.

When to fix it and when to leave it

A short outage that is already over does not need a change, but check the Crawl Stats report to confirm it ended. Do not ignore errors on /robots.txt or errors that keep coming back: Google says its indexing pipeline removes URLs that persistently return a server error, and already indexed URLs are kept for a while but eventually dropped (HTTP status codes and Google Search).

AI crawlers and AI search

Google says 5xx and 429 make its crawlers temporarily slow down, in proportion to the number of URLs returning a server error. If you need to slow Google down, Google advises a temporary 503 or 429 (troubleshooting crawling errors) and says not to use 401 or 403 for that (HTTP status codes and Google Search). Search Console only reports Google's requests. To see whether AI crawlers get the same errors, filter your log by their user agents and verify their IP addresses against the operators' published ranges with verify a crawler IP.

Questions

Does a 503 remove my pages from Google?

Not at once. Google keeps already indexed URLs for a while, but says that returning 503 or 429 for more than 2 days will cause Google to drop those URLs from the index.

What if robots.txt returns a 5xx?

Google stops crawling the site for the first 12 hours while it keeps trying to fetch the file, then uses the last good version for up to 30 days. Network and DNS errors on robots.txt count as server errors too.

Why does the page load fine for me?

The error may be intermittent, tied to load, or limited to some clients. Check the timestamps of Googlebot's requests in your log and run URL Inspection's live test.

Related messages

Sources

Google's documentation, each page read on 2026-10-30. Google updates these pages; if its wording has changed since, Google's page is the reference.

Check the URL: robots.txt per bot, HTTP status, noindex and the HTML a crawler gets.Run the checker