ShapBot (Parallel Web Systems): what it is and how to allow or block it
Updated · checked against the operator's documentation on
ShapBot is Parallel's crawler that discovers and indexes websites for Parallel's web APIs. Its robots.txt token is ShapBot.
Facts
- Operator
- Parallel Web Systems
- Category
- AI search crawler
- robots.txt token
ShapBot
- Honors robots.txt?
- Not stated by Parallel. Parallel suggests allowing ShapBot in robots.txt. Its page names no separate robots.txt token, no crawl-delay and no effective date.
- User-agent example
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0
- User agent we logged
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0Most common variant in our own logs, 86 requests, 19 to 25 Oct 2026. Static snapshot, not the live counter below. A logged string is a claim: anyone can send it.
- Official source
- Parallel: Crawler docs
- Published IP ranges
- https://docs.parallel.ai/resources/shapbot.json
- Compared with the source
What it is used for
Parallel says it helps discover and index websites for Parallel's web APIs.
Builds the index an AI assistant searches when it answers. Blocking it can keep you out of that assistant's answers.
What happens if you block it
Per Parallel, restricting it may reduce your visibility in Parallel's APIs and search features.
In our own logs ShapBot fetched 86 pages in one week without requesting robots.txt once. That is an observation on four small sites, not a conclusion about the crawler. Parallel's page does not name a separate token, so ShapBot is the name it asks you to allow. A related user-triggered fetcher called Shap-User is reported by third parties; we did not verify it.
Is it really ShapBot?
Anyone can send ShapBot as a user agent, so a log line proves nothing by itself. The way to tell is the requesting IP address.
- Compare the IP address with the Parallel Web Systems: published IP ranges. A request from outside those ranges is not the operator's crawler.
- Parallel Web Systems signs its requests (Web Bot Auth) and publishes the Parallel: signature key directory. A valid signature shows a request is genuine whatever its IP address.
- Paste any user agent from your logs into the user-agent lookup, or follow the scripts in Fake Googlebots and PHP scanners.
How to allow or block ShapBot
Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.
User-agent: ShapBot Allow: /
User-agent: ShapBot Disallow: /
Does your site allow ShapBot today? The checker evaluates your robots.txt for this and every other bot.Check my site
What we see on our own sites
Live counter, last 30 days: 7 requests on 1 day identified as ShapBot (latest: ). That is a small sample: read it as an example, not as a measurement of the web.
- Content pages: 4
- Home page: 1
- robots.txt: 1
- Sitemaps: 1
Counts come from the websites we operate; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.
Questions about ShapBot
What is ShapBot?
Parallel's crawler that discovers and indexes websites for Parallel's web APIs. Parallel says it helps discover and index websites for Parallel's web APIs.
Is ShapBot safe and legitimate?
The real ShapBot is an AI search crawler run by Parallel Web Systems. A user-agent string is only a claim, though, so anyone can send it. Compare the requesting IP address with the published IP ranges. Parallel Web Systems signs its requests (Web Bot Auth) and publishes the public keys; a valid signature shows a request is genuine whatever its IP address.
Does ShapBot obey robots.txt?
Not stated by Parallel. Parallel suggests allowing ShapBot in robots.txt. Its page names no separate robots.txt token, no crawl-delay and no effective date.
How do I block ShapBot?
Add a group to the robots.txt at the root of your domain: User-agent: ShapBot followed by Disallow: /. A group that names the bot overrides a general User-agent: * group. Parallel Web Systems documents nothing about robots.txt for this agent, so we cannot promise it is honored.
Will blocking ShapBot hurt my Google rankings or AI visibility?
It can. This bot builds the index an AI assistant searches when it answers, so blocking it can keep you out of that assistant's answers. Per Parallel, restricting it may reduce your visibility in Parallel's APIs and search features.
Related bots
Other AI search crawlers
- OAI-SearchBot OpenAIOpenAI's crawler that surfaces websites in ChatGPT's search features.
- Claude-SearchBot AnthropicAnthropic's crawler that analyzes online content to improve the quality of Claude's search results.
- PerplexityBot PerplexityPerplexity's crawler that surfaces and links websites in Perplexity search results.
- ExaSearchBot ExaExa's crawler that discovers and indexes public pages for its search and retrieval service.
All AI Search Crawlers · Bot directory · Which bots to allow or block · robots.txt generator · See a page as a bot