Crawlerpeek CLI, MCP Server and GitHub Action

The crawlerpeek package is a free, MIT-licensed command-line tool, MCP server and GitHub Action that checks, for each AI crawler, what robots.txt says about your URL and what the page answers when it is fetched with that bot's user-agent string. It needs no account and no API key.

Version 0.2.0 · Node.js 20 or newer · no runtime dependencies · npm · GitHub · By the Crawlerpeek editorial team · Updated 2026-10-25

Install in your editor

Adds the MCP server (npx -y crawlerpeek mcp) to your assistant.

Install in VS CodeVS Code Insiders

Copy

claude mcp add crawlerpeek -- npx -y crawlerpeek mcp

Copy

codex mcp add crawlerpeek -- npx -y crawlerpeek mcp

Claude Desktop: download the .mcpb bundle from the latest GitHub release.

Skills for Claude, Codex, OpenClaw and Hermes: crawlerpeek skill on GitHub.

How do I run it with npx?

Run npx crawlerpeek check <url>; nothing needs to be installed. The default run checks 15 bots: the main AI crawlers plus Googlebot and Bingbot.

npx crawlerpeek check https://example.com

$ npx crawlerpeek check https://example.com
https://example.com
robots.txt: missing (HTTP 404) at https://example.com/robots.txt

Bot                 robots    HTTP
GPTBot              allowed   200
OAI-SearchBot       allowed   200
ClaudeBot           allowed   200
PerplexityBot       allowed   200
Google-Extended     allowed   -
Googlebot           allowed   200
... (15 bots in the default run)

Google-Extended and Applebot-Extended are robots.txt control tokens, not crawlers, so they get a robots.txt decision and no page request. More commands:

npx crawlerpeek check https://example.com --bots GPTBot,ClaudeBot --json
npx crawlerpeek check https://example.com --no-http          # robots.txt only
npx crawlerpeek see https://example.com --bot ClaudeBot      # one bot, full detail
npx crawlerpeek robots --disallow /admin/ --sitemap https://example.com/sitemap.xml
npx crawlerpeek verify 66.249.66.1 --bot Googlebot            # is this address really the crawler?

The verify command checks an IP address from your logs against the operator's published IP ranges or documented reverse DNS. The same check runs in the browser on the crawler IP checker.

How do I fail a CI job when an AI bot is blocked?

Add --fail-on blocked and the command exits with 1 when a bot is shut out, which stops the pipeline.

$ npx crawlerpeek check https://example.com/pricing --bots GPTBot,ClaudeBot,PerplexityBot,Googlebot,Google-Extended --fail-on blocked
https://example.com/pricing
robots.txt: ok (HTTP 200) at https://example.com/robots.txt

Bot              robots    HTTP
GPTBot           blocked   200               FAIL
ClaudeBot        allowed   403 (block page)  FAIL
PerplexityBot    allowed   200
Googlebot        allowed   200
Google-Extended  allowed   -
2 bot(s) failed --fail-on blocked: GPTBot, ClaudeBot
$ echo $?
1

A robots.txt that cannot be fetched (timeout, 5xx, a refused redirect) is reported as unknown and never counts as blocked. Crawlers may treat an unreachable robots.txt as closed, so read unknown as a problem to fix. A full workflow and how to read each failure are in the guide Check AI Crawler Access in CI with GitHub Actions.

How do I use the GitHub Action?

Add uses: crawlerpeek/crawlerpeek/action@v0 to a workflow step with a url, and the job fails when the fail-on condition is met. Inputs: url (required), bots, fail-on (default blocked) and version. The JSON result is the output report.

name: ai-crawler-access
on:
  schedule: [{ cron: "0 6 * * 1" }]
  workflow_dispatch:
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: crawlerpeek/crawlerpeek/action@v0
        with:
          url: https://example.com
          bots: GPTBot,OAI-SearchBot,ClaudeBot,PerplexityBot
          fail-on: blocked

How do I connect the MCP server to an AI assistant?

Start it with npx -y crawlerpeek mcp; any MCP client that can launch a stdio command can use it. The server offers three read-only tools, check_ai_access, see_as_bot and generate_robots_txt, and caches results for 10 minutes. It is listed in the official MCP Registry as io.github.crawlerpeek/crawlerpeek (published 25 October 2026). In Claude Code:

claude mcp add crawlerpeek -- npx -y crawlerpeek mcp

In Claude Desktop, Cursor and other clients that read an mcpServers block, add this to the client's MCP configuration file:

{ "mcpServers": { "crawlerpeek": { "command": "npx", "args": ["-y", "crawlerpeek", "mcp"] } } }

File locations, three example prompts and the answers they return are in Check AI crawler access from Claude Code or Cursor.

Can I use it as a library?

Yes: npm install crawlerpeek gives you checkAiAccess, seeAsBot, generateRobotsTxt and failingBots, as ESM and CommonJS with TypeScript types.

import { checkAiAccess, failingBots } from "crawlerpeek";

const report = await checkAiAccess("https://example.com/pricing", { bots: ["GPTBot", "ClaudeBot"] });
console.log(failingBots(report, "blocked")); // e.g. ["ClaudeBot"]

What does it not do?

It does not show what the real bot receives, because it sends the bot's user-agent string from your machine or runner, not from the operator's IP ranges.

• A site that verifies bot IPs may block a request that only claims the name, so a 403 for Googlebot can be normal. A 403 for one unusual bot next to a 200 for a browser points at a firewall or CDN rule.
• It fetches raw HTML. No JavaScript runs, so it does not show content a browser adds later.
• robots.txt is a request, not access control. Some user-triggered fetchers, such as ChatGPT-User and Perplexity-User, may not follow it; the operators' documentation says which.
• Its bot list comes from the operators' documentation and can drift. The same catalogue is documented on the bot pages.
• It makes one request per bot plus one for robots.txt, with a pause between them, and refuses private and loopback addresses unless you pass --allow-private.

Why did we build it?

We built it because a robots.txt check does not show whether a bot gets through the firewall, and because borrowed bot names are common. On four small sites we operate, 7,428 requests between 19 and 25 October 2026 named one of 11 known bots; 7,096 of them (95.5 percent) came from the operator's published IP ranges. One hosting network alone sent 287 of the 332 requests that fell outside the lists, under 8 different bot names. The numbers and their limits are in the fake bot study.

That finding sets the limit of the tool: a check from a CI runner uses the bot's name but not its addresses, so it can show that a rule shuts a bot name out, and it cannot show how a firewall treats the real crawler. The README and the output say so.

What changed in the latest versions?

Version 0.2.0 was released 2026-10-25, together with 0.1.1 to 0.1.3 (the full changelog is on GitHub). Version 0.2.0 added:

• The verify command and the MCP tool verify_bot_ip, which check an IP address against the operator's published ranges or documented reverse DNS.
• An agent skill for Claude Code, Claude, Codex, OpenClaw and Hermes.
• A Claude Code plugin and marketplace that install the MCP server and the skill together.
• A Claude Desktop extension (.mcpb bundle) attached to each release, which needs no separate Node.js install.
• One-click install links for Cursor and VS Code, plus install commands for Claude Code and Codex.

Versions 0.1.2 and 0.1.3 fixed these things:

• robots.txt length limits count each percent-encoded octet as one character, so a long rule with non-ASCII characters is no longer ignored after normalization (0.1.3).
• A crafted page can no longer make the title extraction run for tens of seconds.
• A redirect to a refused target (private address, bad scheme or port) no longer aborts the whole run. It is reported for that bot, or as unknown for robots.txt, with the reason.
• robots.txt paths are compared after percent-encoding normalization (RFC 9309), so rules such as Disallow: /ü or /%7Efoo match the way crawlers read them.
• An unknown option is an error (exit code 2), so a typo like --failon cannot silently pass a CI run.
• The Applebot user agent is the full string from Apple's documentation.

Questions about the crawlerpeek package

Prefer a browser? The checker runs the robots.txt part on the web, and See as bot shows a page without JavaScript. Source and issues: GitHub; releases: npm.