Check AI crawler access from Claude Code or Cursor

What does the crawlerpeek MCP server do?

The crawlerpeek MCP server is a local program that speaks the Model Context Protocol over stdio and gives an assistant three read-only tools. The tools only fetch the URL you name and evaluate its robots.txt; no model is called. Results of the two fetching tools are cached for 10 minutes per call.

The server is also listed in the official MCP Registry as io.github.crawlerpeek/crawlerpeek, published on 25 October 2026. The package itself is described on the CLI and MCP page.

How do I add it to Claude Code?

Run claude mcp add crawlerpeek -- npx -y crawlerpeek mcp in a terminal. The -- separates Claude Code's own options from the command that starts the server, and everything after it is passed to the server untouched (Claude Code MCP documentation, checked 25 October 2026).

claude mcp add crawlerpeek -- npx -y crawlerpeek mcp

How do I add it to Cursor?

Add the server to .cursor/mcp.json in your project folder, or to ~/.cursor/mcp.json to make it available in every project (Cursor MCP documentation, checked 25 October 2026).

{
  "mcpServers": {
    "crawlerpeek": {
      "command": "npx",
      "args": ["-y", "crawlerpeek", "mcp"]
    }
  }
}

How do I add it to Claude Desktop?

Open Settings, then Developer, then Edit Config, and add the same mcpServers block to claude_desktop_config.json. The file is at ~/Library/Application Support/Claude/ on macOS and %APPDATA%\Claude\ on Windows, and Claude Desktop must be quit completely and restarted to load it (Model Context Protocol documentation, checked 25 October 2026).

What can I ask the assistant to check?

You can ask in plain language and the assistant picks the tool. Three prompts cover all three tools; the outputs below are shortened from a real run of crawlerpeek 0.1.2 against example.com on 25 October 2026.

Prompt 1: “Can GPTBot and ClaudeBot read https://example.com/?” The assistant calls check_ai_access with url and bots. The server returns one entry per bot, which the assistant summarizes:

{
  "robotsTxt": { "state": "missing", "status": 404, "error": null },
  "bots": [
    { "bot": "GPTBot",    "robots": { "access": "allowed" }, "http": { "status": 200, "outcome": "ok", "title": "Example Domain" } },
    { "bot": "ClaudeBot", "robots": { "access": "allowed" }, "http": { "status": 200, "outcome": "ok", "title": "Example Domain" } }
  ]
}

Here robots.txt is missing (HTTP 404), which means every bot is allowed, and both bots got a 200. Prompt 2: “What does PerplexityBot get when it fetches the homepage?” The assistant calls see_as_bot:

{
  "bot": "PerplexityBot", "status": 200, "finalUrl": "https://example.com/", "redirects": 0,
  "contentType": "text/html; charset=utf-8", "bytes": 577, "title": "Example Domain",
  "looksBlocked": false, "outcome": "ok"
}

Prompt 3: “Write a robots.txt that closes CCBot, keeps /admin/ private and lists my sitemap.” The assistant calls generate_robots_txt with disallow, block_bots and sitemap. The text below is shortened; the real output names 15 bots in the first group.

User-agent: GPTBot
User-agent: OAI-SearchBot
... (13 more bots)
Allow: /
Disallow: /admin/

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

The generated file repeats Disallow: /admin/ in the bot group on purpose: a bot with its own group ignores the * group (how groups are chosen).

What are the limits of checking through an assistant?

The result is only as good as the check underneath it, and the check sends each bot's user-agent string from your machine, not from the operator's IP ranges. A site that verifies bot IP addresses may answer the real bot differently; our study of forged bot names explains why that verification exists.

• A 403 for Googlebot can be correct behavior, because a firewall may refuse a request that only claims the name. A 403 for one unusual bot next to a 200 for the others points at a rule you did not intend.
• The tools fetch raw HTML and run no JavaScript, and robots.txt is a request, not access control.
• After you fix something, the same call can return the cached answer for up to 10 minutes. Wait, or restart the server, before you re-check.
• Private, loopback and metadata addresses are refused, so a local dev server cannot be checked unless you start the server with --allow-private.
• The assistant explains the result; read the returned status and robots values yourself before you change a firewall rule.

To get a failed job instead of a chat answer when a deploy shuts a bot out, run the same check in CI: check AI crawler access with GitHub Actions.

What did we check, and when?

We checked on 25 October 2026. Tool names, arguments and outputs are from crawlerpeek 0.1.2 (source, npm). Client setup is from the Claude Code, Cursor and Model Context Protocol documentation. Clients change their menus and file locations, so check the originals before you rely on a detail.