Crawlerpeek
Check a site

Blocked due to access forbidden (403): when your server or firewall turns Googlebot away

Your server answered Googlebot with 403 Forbidden. Googlebot never sends credentials, so let it in: check firewall, bot protection and login rules for the URL.

The message in Search Console

Blocked due to access forbidden (403)

Page indexing report, group "Not indexed". Spelling as in Google's help page for the report.

By the Crawlerpeek editorial team · Sources checked · Updated · Editorial standards

What Google says

HTTP 403 means that the user agent provided credentials, but was not granted access. However, Googlebot never provides credentials, so your server is returning this error incorrectly. The page will not be indexed.

Source: Google Search Console Help: Page indexing report, read 2026-10-30.

Common causes

  • A firewall, bot protection or CDN rule that blocks Googlebot by user agent, IP range, country or request rate.
  • A 403 used to slow crawlers down. Google says not to use 401 and 403 status codes for limiting the crawl rate (HTTP status codes and Google Search).
  • Pages behind a login or paywall that answer 403 to anyone without a session.
  • Server permissions on files or directories, or hotlink and referrer rules that also catch crawlers.

How to check

  1. Run URL Inspection's live test in Search Console: it shows what Google itself gets now.
  2. In your access log, find the 403 answers to requests that say Googlebot and confirm the IP addresses with verify a crawler IP. Google documents two ways to verify: reverse DNS to googlebot.com, google.com or googleusercontent.com with a forward lookup, or a match against its published IP ranges (Verifying Googlebot).
  3. Enter the URL in the AI crawler checker. It requests the page with a browser user agent and never as Googlebot, so a 403 there means the block hits ordinary clients too; a 200 there does not rule out a rule aimed at Googlebot.
  4. Allow verified Googlebot requests in the firewall or bot rules, then click Validate fix.

When to fix it and when to leave it

Leave it for pages you do not want in Google, such as member areas: Google says the page will not be indexed. If you do want it indexed, Google says to admit non-signed-in users or explicitly allow Googlebot requests without authentication, after verifying its identity. A 403 on /robots.txt is a separate trap: Google treats all 4xx errors except 429 as if no robots.txt file existed, so your rules stop applying (Google's robots.txt specification).

AI crawlers and AI search

Bot protection does not only hit Googlebot. In our own check of 151 large websites run by others (29 October 2026), 9 answered our request for /robots.txt with HTTP 403; our client named itself as Crawlerpeek, and bot protection is the likely cause (who blocks AI crawlers). Under Google's rules a 403 on robots.txt means no crawl restrictions, and a crawler that gets a 403 for the file cannot read your rules at all. To let a specific AI crawler through, verify it by the IP ranges its operator publishes: verify a crawler IP checks an address against them, and the bot directory shows how each bot can be verified.

Questions

Why does Search Console say 403 when the page opens in my browser?

Your browser sends a browser user agent and often cookies or a login session, and firewall rules may treat it differently from a crawler. Google says Googlebot never provides credentials. URL Inspection's live test shows what Google gets.

Does a 403 affect my other pages?

Google says 4xx codes other than 429 have no effect on crawl rate. The affected URL is removed from the index if it was indexed.

What is the difference to a 401?

Search Console reports 401 separately as "Blocked due to unauthorized request (401)": the page asked for authorization. A 403 means access was refused.

Related messages

  • Server error (5xx) · Your server answered Googlebot with a 500-level status, so the page was not indexed. Find the failing requests in your logs, fix the cause, then validate.
  • URL blocked by robots.txt · A Disallow rule in your robots.txt stops Googlebot from crawling this URL. If the page should be in Google, remove or narrow that rule and validate the fix.
  • Soft 404 · The page returns 200 OK but looks like an error or empty page to Google. Send a real 404 or 410 if it is gone, a 301 if it moved, or fix what failed to load.
  • Search Console Crawl and Indexing Errors

Sources

Google's documentation, each page read on 2026-10-30. Google updates these pages; if its wording has changed since, Google's page is the reference.

Check the URL: robots.txt per bot, HTTP status, noindex and the HTML a crawler gets.Run the checker