SemrushBot (Semrush): what it is and how to allow or block it
SemrushBot is Semrush's crawler for data behind tools such as Backlink Analytics and Site Audit. Its robots.txt token is SemrushBot.
Facts
- Operator
- Semrush
- Category
- SEO tool crawler
- robots.txt token
SemrushBot
- Honors robots.txt?
- Yes, per Semrush. Follows robots.txt. A 4xx answer means no restrictions apply; a 5xx answer blocks crawling of the whole site. Semrush says it can take up to one hour or 100 requests to notice robots.txt changes. Crawl-delay is supported up to 10 seconds.
- User-agent example
- Contains
semrushbot(the operator documents no full example string)
- Official source
- Semrush: SemrushBot
- Compared with the source
What it is used for
Discovers and collects new and updated web data for tools such as Backlink Analytics, Site Audit and Backlink Audit. Semrush lists further tokens for other tools, for example SiteAuditBot and SemrushBot-BA. Crawls the web for a marketing or SEO tool, mostly to build a backlink or site database. It does not decide whether you rank, and blocking it only removes you from that tool's data.
What happens if you block it
Use the matching token with Disallow: /. Semrush says not to block by IP address, because it does not use consecutive IP ranges.
How to allow or block SemrushBot
Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.
User-agent: SemrushBot Allow: /
User-agent: SemrushBot Disallow: /
Does your site allow SemrushBot today? The checker evaluates your robots.txt for this and every other bot.Check my site
What we see on our own sites
No request identifying as SemrushBot has been recorded in the last 30 days, so there is nothing to show yet. We never fill this in with estimates.
Counts come from the websites we operate; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.
Related bots
Other SEO tool crawlers
- AhrefsBot AhrefsThe crawler of Ahrefs, a marketing intelligence platform; its data also powers the Yep search engine.
- MJ12bot MajesticMajestic's crawler that maps link relationships between websites for its search engine and backlink index.
All SEO Tool Crawlers · Bot directory · Which bots to allow or block · robots.txt generator · See a page as a bot