Check AI crawler access in CI with GitHub Actions

Why does AI crawler access break without anyone noticing?

AI crawler access breaks silently because nothing on your side reports an error when a robots.txt change or a firewall rule starts refusing a bot. Typical causes: a staging robots.txt with Disallow: / ships to production; a firewall rule added for scrapers also matches ClaudeBot; a bot-management setting starts answering unknown clients with a challenge page. The site still loads in your browser, nothing alerts, and the first sign is that an AI assistant stops citing you. The common robots.txt mistakes and the Cloudflare rules that block AI bots are the usual causes.

A check in CI turns that into a failed job. It tests two things per bot: what robots.txt says about the URL you give it, and what the page answers when it is fetched with that bot's user-agent string.

What does a working GitHub Actions workflow look like?

This workflow runs crawlerpeek check every Monday at 06:00 UTC and on demand, and fails the job when a bot is blocked. It uses the Crawlerpeek GitHub Action, which calls the open-source crawlerpeek package. List the bots you care about; without bots it checks 15 of the main AI crawlers plus Googlebot and Bingbot. To run the same check from your editor instead, see the MCP setup for Claude Code and Cursor.

name: ai-crawler-access
on:
  schedule: [{ cron: "0 6 * * 1" }]
  workflow_dispatch:
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: crawlerpeek/crawlerpeek/action@v0
        with:
          url: https://example.com
          bots: GPTBot,OAI-SearchBot,ClaudeBot,Claude-SearchBot,PerplexityBot
          fail-on: blocked

Save it as .github/workflows/ai-crawler-access.yml. If you prefer no action, replace the step with npx crawlerpeek check https://example.com --bots GPTBot,ClaudeBot --fail-on blocked; the exit code is 1 when the condition is met. The options are listed on the CLI page.

How do I read a failed check?

Each failing row says whether robots.txt or the page response closed the door. The log prints one row per bot with the robots.txt decision and the HTTP result, and marks failing rows. This is output from a test server that blocks GPTBot in robots.txt and challenges ClaudeBot:

Bot              robots    HTTP
GPTBot           blocked   200               FAIL
ClaudeBot        allowed   403 (block page)  FAIL
PerplexityBot    allowed   200
Googlebot        allowed   200
Google-Extended  allowed   -
2 bot(s) failed --fail-on blocked: GPTBot, ClaudeBot

With fail-on: any-non-200 the job also fails on a 404, a 5xx, a redirect that ends elsewhere or a failed request. Use it for a page that must always answer 200; for a whole-site watch, blocked produces fewer false alarms.

Should the check run on a schedule or after each deploy?

Do both: a schedule catches changes nobody deployed, and a run after each deploy catches the ones you shipped.

• Schedule: Catches changes you did not deploy: a CDN rule edited in a dashboard, a plugin that rewrites robots.txt, a setting changed by someone else. GitHub runs scheduled workflows in UTC, only from the default branch, allows an interval of 5 minutes at the shortest, and may delay them under high load. In a public repository, scheduled workflows are disabled after 60 days without repository activity (GitHub documentation).
• After a deploy: Catches changes you shipped. Do not trigger on push alone: the job can run before the new version is live and test the old one. Trigger on the workflow that deploys, with workflow_run, so the check starts when that workflow has finished.

on:
  workflow_run:
    workflows: ["Deploy"]
    types: [completed]

jobs:
  check:
    if: github.event.workflow_run.conclusion == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: crawlerpeek/crawlerpeek/action@v0
        with:
          url: https://example.com
          fail-on: blocked

Replace Deploy with the name of your deploy workflow. A CDN can serve the previous robots.txt from its cache for a while, so a failure right after a deploy is worth repeating once before you act.

What can a CI check not tell you?

It cannot tell you how your firewall treats the real crawler's IP addresses. The check sends each bot's user-agent string from a GitHub runner. It is not the real bot, and the request does not come from the operator's published IP ranges. A firewall that verifies bot IP addresses may treat a request that only claims to be Googlebot differently from the real crawler, usually by blocking it. Two consequences:

• A 403 for Googlebot or GPTBot in CI can be correct behavior: your firewall recognises a forged name. A 403 for an unusual bot next to a 200 for the others, or a challenge page for every bot, points at a rule you did not intend.
• A pass does not prove the real bot gets through. A firewall that lets the real GPTBot in by IP address, or one that blocks it by IP address, is not visible from the runner. Your server logs are the evidence: see fake crawlers and scanners for how to compare requests with the operators' lists, and our study of forged bot names for how common a borrowed name is.

The check also fetches raw HTML and runs no JavaScript, and robots.txt is a request, not access control. Some user-triggered fetchers, such as ChatGPT-User and Perplexity-User, may not follow it; the operators' documentation says which.

What did we check, and when?

We checked on 25 October 2026. Behavior of the CLI and action is from crawlerpeek 0.1.2 (npm, GitHub). Scheduling details are from GitHub's documentation of workflow triggers. Both can change, so check the originals before you rely on a detail.