Crawlerpeek
Check a site

Retry-After Header for Crawlers: What Each Bot Documents

Google documents Retry-After on 429 and 503 for its crawlers (guide updated 6 Oct 2026). Bing, OpenAI, Anthropic, Perplexity compared, plus a tested demo.

Published
8 min read
  • retry-after
  • http-429
  • http-503
  • crawl-rate
  • googlebot
  • crawl-delay

The Retry-After header tells a client when to try again after a 429 or 503, and Google documents it for its crawlers: a delay in seconds or a UTC date, with examples added to its crawl-rate guide on 6 October 2026. Bing, OpenAI, Anthropic and Perplexity say nothing about it on the pages we read.

As of 1 November 2026.

Picture a log excerpt from a small shop. A crawler asks for filter pages at a rate your database cannot carry, and you decide to push back. The lines below are a made-up example, with an address from the documentation range.

203.0.113.7 - - [01/Nov/2026:09:14:02 +0000] "GET /shop?color=red&size=m HTTP/1.1" 200 5120
203.0.113.7 - - [01/Nov/2026:09:14:02 +0000] "GET /shop?color=red&size=l HTTP/1.1" 200 5120
203.0.113.7 - - [01/Nov/2026:09:14:03 +0000] "GET /shop?color=blue&size=m HTTP/1.1" 200 5120

You can answer with a 429, a 503, a robots.txt Crawl-delay, or all three. Whether the bot slows down depends on what its operator has written down, and the operators have written very different amounts. So who documents what?

Which crawler operators document 429, 503 and Retry-After?

Only Google documents 429, 503 and Retry-After for its crawlers. Anthropic and Bing document Crawl-delay, Google says it does not support that field, and the OpenAI and Perplexity crawler pages we read mention none of the four. The table lists what each operator's own page says, with a cell marked "not found" where we read the page and the term is absent.

Method. On 1 November 2026 we fetched each page below with curl and a browser user agent, stripped the HTML, and searched the text for retry-after, 429, 503, too many requests and crawl-delay (match counts are saved with our test files). "Not found" means zero matches on that page. It does not mean the operator never throttles by other means, and it does not cover pages we did not read. For Bing we read four current Webmaster help articles (crawler overview, Crawl control, robots.txt, reporting a Bingbot issue), which the help center serves through a content endpoint; the August 2009 MSNBot blog post is kept only as dated background.

Operator (crawler) 429 or 503 Retry-After Crawl-delay Source page and date
Google (Googlebot and other Google crawlers) Slows crawling; 429 treated as a server error Documented for 503 and 429, seconds or UTC date Not supported Crawl-rate guide, updated 2026-10-06
Microsoft (Bing, Bingbot) Not found Not found Documented in robots.txt; Crawl control tool for hourly rates Bing Webmaster help articles, read 2026-11-01 (no page dates shown)
OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User) Not found Not found Not found OpenAI crawlers page, read 2026-11-01
Anthropic (ClaudeBot and others) Not found Not found "respecting Crawl-delay where appropriate" Help Center article, dated April 7, 2026
Perplexity (PerplexityBot, Perplexity-User) Not found Not found Not found Perplexity crawlers guide, read 2026-11-01

Notice what the table does not say. A "not found" is not "ignored". It means the operator's crawler page, read on 1 November 2026, does not mention the term, so you cannot cite the operator's own documentation if you rely on that signal. Bing's Crawl control help adds that a crawl-delay in robots.txt "will always take precedence" over the hourly settings. The next question is what Google's cells actually promise.

What does Googlebot do when it sees a 429 or 503?

Google's crawl-rate guide says that when its infrastructure meets a significant number of URLs with 500, 503 or 429 responses, it reduces the crawl rate across the whole hostname and raises it again once the errors drop. The reduction covers URLs that return content too, not only the ones that return errors.

Google's page on HTTP status codes adds that crawlers "treat the 429 status code as a signal that the server is overloaded, and it's considered a server error." For Search, URLs that keep returning errors eventually drop out of the index. The crawl-rate guide does not recommend doing this for longer than 1-2 days: if Googlebot sees these codes on the same URL "for multiple days, the URL may be dropped from Google's index."

Google's changelog entry for 6 October 2026 says the Retry-After support "is not new": the header was already documented in the page on pausing or disabling a website. What changed on that date is the placement: the examples sit in the crawl-rate guide. Google documents two value forms there.

HTTP/1.1 503 Service Unavailable
Retry-After: 120

HTTP/1.1 503 Service Unavailable
Retry-After: Wed, 21 Oct 2026 07:28:00 GMT

Google's guide says the header indicates "when Google's crawlers can retry the request". It does not publish a promise that a crawler waits the full interval, so treat the header as a hint and keep the status code as the real signal. How does the header look on the wire, and what do the two forms parse to?

How do I send a 429 or 503 with a Retry-After header? A tested demo

A 429 with Retry-After: 120 is a few lines of code. We ran a Python http.server handler on localhost:8890 with three routes: /busy returns 429 with a seconds value, /maintenance returns 503 with an HTTP-date one hour ahead, and /bare503 returns 503 with no header. We fetched each with curl and a Googlebot user agent string.

from http.server import BaseHTTPRequestHandler, HTTPServer
from email.utils import formatdate
import time

class H(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/busy":
            code, hdr = 429, {"Retry-After": "120"}
        elif self.path == "/maintenance":
            code, hdr = 503, {"Retry-After": formatdate(time.time() + 3600, usegmt=True)}
        elif self.path == "/bare503":
            code, hdr = 503, {}
        else:
            code, hdr = 200, {}
        body = b"ok\n" if code == 200 else b"try later\n"
        self.send_response(code)
        for k, v in hdr.items():
            self.send_header(k, v)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

HTTPServer(("localhost", 8890), H).serve_forever()

Output of curl -si -A 'Googlebot/2.1' http://localhost:8890/<route> for the three routes, run on 1 November 2026 at 19:41 UTC, with the Server, Date and Content-Length lines left out:

HTTP/1.0 429 Too Many Requests
Retry-After: 120

HTTP/1.0 503 Service Unavailable
Retry-After: Sun, 01 Nov 2026 20:41:31 GMT

HTTP/1.0 503 Service Unavailable

The header is a plain response header, so it works the same behind nginx, Apache or a CDN rule, as long as that layer passes it through. We did not test those layers. A client that wants to honor the header must parse both forms. Python's standard library handles it:

from datetime import timedelta
from email.utils import parsedate_to_datetime

def next_try(value, now):
    value = value.strip()
    if value.isdigit():
        return now + timedelta(seconds=int(value))
    return parsedate_to_datetime(value)

With now set to 19:41:31 UTC, 120 gave 19:43:31 and the HTTP-date gave 20:41:31, which is +3600 seconds. This demo proves what the server sends and how the two forms parse. It does not prove what any crawler does with the response, because no crawler requested our test server. That part is documented only where the table says so.

Which status code should you reach for?

Should I use 429, 503 or Crawl-delay?

Use 429 or 503 with Retry-After when you need relief within hours, and use Crawl-delay only for operators whose documentation mentions it. Google documents the status codes and does not support the field; Anthropic and Bing document the field. For the rest, the operator pages we read promise neither.

The standards give the status codes distinct meanings. RFC 9110 defines 503 as a server "unable to handle the request due to a temporary overload or scheduled maintenance" and says the server MAY send Retry-After. RFC 6585 defines 429 as the user having "sent too many requests in a given amount of time", with an optional Retry-After. The same RFC adds that "servers are not required to use the 429 status code", and that under attack it "may be more appropriate to just drop connections".

Two cautions from Google's own pages. Do not return 503 for your robots.txt file, because Google's page on disabling a site says this "blocks all crawling". And keep the emergency short. For crawlers you want gone for good, block AI crawlers in robots.txt or at the server instead of throttling them forever.

Before you rate-limit by user agent, check that the address is really the operator's. Anyone can send the string Googlebot, so an unverified limit may punish the wrong party. The Google operator page and the Bingbot entry are the places to start, and the verify bot IP tool tests an address against an operator's published ranges.

Common follow-up questions

Should I send 429 or 503 to Googlebot?

Google documents both status codes with Retry-After for its crawlers, in the crawl-rate guide updated 6 Oct 2026. Use either for short relief, keep the emergency brief, and never return 503 for robots.txt, because that blocks all crawling.

Does Googlebot obey Retry-After?

Google documents the header for 503 and 429 responses and says it indicates when its crawlers can retry. The page does not state how long a crawler waits, and our demo server received no Googlebot request, so we make no claim about observed behavior.

Does GPTBot respect 429 or Crawl-delay?

OpenAI's crawlers page, read on 1 November 2026, contains neither term. The page documents robots.txt tags and published IP ranges (see the OpenAI operator page). Without a documented behavior, measure your own logs instead of assuming, and check that the requests are real with the GPTBot IP ranges method.

Does ClaudeBot support Crawl-delay?

Yes, according to Anthropic's Help Center article dated April 7, 2026, which says it supports the non-standard Crawl-delay extension and gives User-agent: ClaudeBot with Crawl-delay: 1 as an example. The same article does not mention 429, 503 or Retry-After.

Why is my site missing from Google after a long 503?

Google's crawl-rate guide warns that if Googlebot sees these status codes on the same URL for multiple days, the URL may be dropped from the index. If crawling stalled for another reason, why Google isn't crawling walks through the causes.

Does Perplexity document any throttling signal?

Not on its crawlers guide, read on 1 November 2026. The guide covers robots.txt tags, firewall configuration and IP address sources, and none of 429, 503, Retry-After or Crawl-delay appears.

Sources

Checked 1 November 2026.