YandexBot
YandexBot is Yandex's crawler for building the index its search and answers draw on.
Should you let it in?
Allow or block YandexBot
Yes, if you want to appear in Yandex's search and answers. Blocking it can keep your pages out of them.
To ask it to stay out, add this to robots.txt. Whether it follows robots.txt is Yandex's call; its documentation says.
User-agent: YandexBot
Disallow: /Is it really?
How to tell a real YandexBot from an impostor
Anyone can put “YandexBot” in a user agent. What can't be faked:
- Check reverse DNS. Look up the requesting IP's host name; it should end in
yandex.ruoryandex.netoryandex.com. Then look that name up and make sure it resolves back to the same IP, since anyone can set a reverse DNS name on their own addresses.
In Punchline
How Punchline counts YandexBot
Punchline labels a request that names YandexBot with how sure it is: verified or declared when the evidence checks out, a lower tier when it can't be checked, and impostor when the evidence contradicts it. Each visit is counted under Yandex, with its purpose, so you can see whether YandexBot is reading your site and which pages it reads. How it works · Known agents