Crawlerpeek
Check a site

YaDirectFetcher (Yandex): what it is and how to allow or block it

Updated · checked against the operator's documentation on

YaDirectFetcher is Yandex's ad robot that checks pages that ads link to. Verify it by reverse DNS; per Yandex it ignores robots.txt, so a block will not hold. Its robots.txt token is YaDirectFetcher.

Facts

Operator
Yandex
Category
Special-purpose crawler
robots.txt token
YaDirectFetcher
Honors robots.txt?
No, per Yandex. Yandex says YaDirectFetcher does not use the robots.txt file and ignores the directives set for it.
User-agent example
Mozilla/5.0 (compatible; YaDirectFetcher/1.0; Dyatel; +http://yandex.com/bots)
Official source
Yandex: Verifying Yandex robots
Compared with the source

What it is used for

Per Yandex, YaDirectFetcher downloads target pages of ads to check their availability and topic, which is needed for ad placement in search results and on partner sites.

Not the crawler behind a search index. What blocking it changes is specific to its tool, so read the next section.

What happens if you block it

Per Yandex, a robots.txt rule changes nothing, because YaDirectFetcher does not use the robots.txt file and ignores the directives set for it.

Yandex says its robots use networks AS13238, AS208722 and AS212066 with addresses that change often, so it does not disclose a list and recommends the reverse DNS check.

Is it really YaDirectFetcher?

Anyone can send YaDirectFetcher as a user agent, so a log line proves nothing by itself. The way to tell is the requesting IP address.

How to allow or block YaDirectFetcher

Add one of these groups to the robots.txt at the root of your domain. A group that names the bot overrides a general User-agent: * group.

Allow YaDirectFetcher
User-agent: YaDirectFetcher
Allow: /
Block YaDirectFetcher
User-agent: YaDirectFetcher
Disallow: /

Does your site allow YaDirectFetcher today? The checker evaluates your robots.txt for this and every other bot.Check my site

What we see on this site

No request identifying as YaDirectFetcher has been recorded since 19 Oct, so there is nothing to show yet. We never fill this in with estimates.

Counts come from this site's own server; they hold no IP addresses or URLs, and a user agent is only a claim. See the bot traffic data and the methodology.

Questions about YaDirectFetcher

What is YaDirectFetcher?

Yandex's robot that downloads the target pages of ads to check their availability and topic. Per Yandex, YaDirectFetcher downloads target pages of ads to check their availability and topic, which is needed for ad placement in search results and on partner sites.

Is YaDirectFetcher safe and legitimate?

The real YaDirectFetcher is a special-purpose crawler run by Yandex. A user-agent string is only a claim, though, so anyone can send it. Yandex documents a reverse lookup whose name ends in yandex.ru, yandex.net or yandex.com, followed by a forward lookup that must return the same address. It does not publish its robots' IP addresses.

Does YaDirectFetcher obey robots.txt?

No, per Yandex. Yandex says YaDirectFetcher does not use the robots.txt file and ignores the directives set for it.

How do I block YaDirectFetcher?

Robots.txt may not stop it. Yandex says YaDirectFetcher does not use the robots.txt file and ignores the directives set for it. If you add a group anyway, use User-agent: YaDirectFetcher followed by Disallow: /.

Will blocking YaDirectFetcher hurt my Google rankings or AI visibility?

It is not Yandex's indexing robot. Per Yandex, it checks pages that ads link to and ignores robots.txt, so a rule for it changes nothing.

Related bots

More from Yandex

Other Special-purpose crawlers

All Special-Purpose Crawlers · Bot directory · Bots from Yandex · Which bots to allow or block · robots.txt generator · See a page as a bot