How to block AhrefsBot, SemrushBot, MJ12bot and other SEO crawlers

Which robots.txt tokens do these crawlers use?

Each vendor documents its own tokens. Several vendors run more than one crawler, and a rule for one token does not cover the others. Tokens below are copied from the vendor pages, read on 26 October 2026.

The list is not exhaustive. Moz's DotBot has its own bot page with Moz's statements, and every SEO tool crawler we track is listed in the SEO tool crawlers category.

How do I block them in robots.txt?

Add this to the robots.txt at the root of each host. A group that names a bot overrides a general User-agent: * group, so it works even if your default is open. Subdomains need their own file; Semrush says so explicitly.

User-agent: AhrefsBot
Disallow: /

User-agent: AhrefsSiteAudit
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: SiteAuditBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: DataForSeoBot
Disallow: /

Semrush lists further tokens for its other tools (SemrushBot-BA, SemrushBot-SI, SemrushBot-SWA, SplitSignalBot, SemrushBot-OCOB, SemrushBot-FT, RyteBot, SemrushBot-ESI). Add a group for each one you want to refuse. SiteAuditBot and AhrefsSiteAudit are the crawlers behind the vendors' site-audit tools, which site owners often run on their own sites. Blocking them stops your own audits in those tools. Ahrefs says AhrefsSiteAudit can ignore robots.txt only when a verified site owner requests it.

How do I slow a crawler down instead of blocking it?

Crawl-delay is a non-standard directive, and only some crawlers read it. Ahrefs, Semrush and Majestic each say they support it, with the limits in the table above.

User-agent: AhrefsBot
Crawl-delay: 10

User-agent: SemrushBot
Crawl-delay: 10

User-agent: MJ12bot
Crawl-delay: 20

Ahrefs adds that it backs off by itself when your server returns a temporary 4xx or 5xx status.

What happens to robots.txt errors?

• Ahrefs says its bots cannot recognise the commands in a robots.txt that contains errors, and keep crawling as before. Check the file with a plain request before you rely on it (how to test robots.txt).
• Semrush: the file must return HTTP 200. A 4xx answer means no restrictions apply, and a 5xx answer stops Semrush from crawling the entire site.
• Majestic: if the file cannot be retrieved, MJ12bot "will assume that it is okay to crawl your site". A 403 Forbidden on robots.txt is handled as a blanket disallow. An MJ12bot group takes precedence over the * group instead of being combined with it.
• Do not apply a server-level block to /robots.txt itself unless you accept those behaviours.

How do I block them at the server?

Use a server rule when a crawler ignores robots.txt or when you want the refusal enforced. It matches the User-Agent header, so it only stops requests that send the real name. A scraper that borrows one of these names is stopped the same way; a scraper that sends another name is not. Keep the robots.txt group as well: it is the documented opt-out.

nginx rule

# http { ... }
map $http_user_agent $block_seo_bot {
    default 0;
    ~*(AhrefsBot|AhrefsSiteAudit|SemrushBot|SiteAuditBot|SplitSignalBot|RyteBot|MJ12bot|DataForSeoBot) 1;
}

# server { ... }
if ($block_seo_bot) {
    return 403;
}

SemrushBot also matches SemrushBot-BA, SemrushBot-SI and the other SemrushBot- tokens. Reload with nginx -t && nginx -s reload and test with curl -I -A "MJ12bot" https://example.com/. The full nginx walkthrough covers map placement.

Apache rule

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (AhrefsBot|AhrefsSiteAudit|SemrushBot|SiteAuditBot|SplitSignalBot|RyteBot|MJ12bot|DataForSeoBot) [NC]
RewriteRule .* - [F,L]

Put it in .htaccess or the virtual host. Apache details.

Cloudflare WAF rule

Create a WAF custom rule with the action Block and this expression. Cloudflare's string operators are case-sensitive, so lower() is used to match any capitalisation (Cloudflare rules language: operators, read 26 October 2026).

(lower(http.user_agent) contains "ahrefsbot") or
(lower(http.user_agent) contains "ahrefssiteaudit") or
(lower(http.user_agent) contains "semrushbot") or
(lower(http.user_agent) contains "siteauditbot") or
(lower(http.user_agent) contains "mj12bot") or
(lower(http.user_agent) contains "dataforseobot")

Cloudflare says a bot needs to obey robots.txt and crawl directives to qualify as a verified bot (Cloudflare: verified bots, read 26 October 2026). For the AI-crawler controls in the same dashboard see block AI crawlers with Cloudflare.

Should I block by IP address?

The three vendors differ, and two advise against it.

• Ahrefs publishes its ranges (IP ranges, individual IPs). Ahrefs links them from its whitelisting help, so they are meant for allowing and verifying the crawler.
• Semrush: "Do not try to block SemrushBot via IP as we do not use any consecutive IP blocks."
• Majestic advises against blocking by IP in .htaccess: the bot is distributed and does not use consecutive IP blocks.

For Semrush and Majestic, robots.txt plus a user-agent rule is the practical route.

How do I check a request is really AhrefsBot, SemrushBot or MJ12bot?

A user agent is only a claim. What you can check depends on what the vendor publishes (pages read 26 October 2026).

So only Ahrefs and DataForSEO can be checked from the log line alone. For Semrush and MJ12bot a request from an unknown network cannot be proven fake or real by IP, and the vendor gives you no list to compare. Check whether the client fetched /robots.txt and whether it follows it.

To verify an Ahrefs address, compare it with the published ranges. This script reads Ahrefs' list (a prefixes array with ipv4Prefix or ipv6Prefix):

# verify_ahrefs.py: is this IP address in Ahrefs' published crawler ranges?
import ipaddress, json, sys, urllib.request

URL = "https://api.ahrefs.com/v3/public/crawler-ip-ranges"
ip = ipaddress.ip_address(sys.argv[1])
req = urllib.request.Request(URL, headers={"User-Agent": "verify-crawler"})
prefixes = json.load(urllib.request.urlopen(req))["prefixes"]
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in prefixes]
print("in list" if any(ip in n for n in nets if n.version == ip.version) else "NOT in list")

Or use the reverse-DNS route Ahrefs documents:

host <ip from your log>
# the name must end in ahrefs.com or ahrefs.net
host <that name>
# the forward lookup must return the same IP again

For other operators, the crawler IP checker runs the list and reverse-DNS checks, and the user-agent lookup names the bot behind a string from your log. In our own logs from 19 to 25 October 2026 (four small sites), all 651 requests that named AhrefsBot came from Ahrefs' published list (study and limits). That is one week on four sites, so it says little about yours. Our PHP scanner log study shows how requests that only borrow a crawler's name look.

What does blocking cost you?

These crawlers feed marketing tools, not search engines. Per each vendor, the data goes into their own products: Ahrefs' link and site database (and the Yep search engine), Semrush's Backlink Analytics and related tools, Majestic's Site Explorer backlink index.

• Your site stops being refreshed in that tool. Links to and from your pages appear late or never in its backlink and keyword reports, so anyone who uses the tool (you, a client, a link-building partner) sees stale data.
• Your own audits fail if you block the site-audit tokens (AhrefsSiteAudit, SiteAuditBot).
• Google and Bing rankings do not depend on these crawlers. Google's crawler overview lists Googlebot as the crawler for Search; it is a different bot. Make sure no rule matches Googlebot or Bingbot.
• Only the blocked tool is affected. Other tools, and any crawler that does not identify itself, are not stopped by these rules.
• A block removes some server load. How much depends on your traffic; we did not measure it on your site and you should check your own logs before deciding.

Checked 26 October 2026. Vendor pages change, so use the tokens from the vendor pages linked above if they differ from this table.