punchline!

SemrushBot

Builds a search or answer indexName only

SemrushBot is Semrush's crawler for building the index its search and answers draw on.

Should you let it in?

Allow or block SemrushBot

Yes, if you want to appear in Semrush's search and answers. Blocking it can keep your pages out of them.

To ask it to stay out, add this to robots.txt. Whether it follows robots.txt is Semrush's call; its documentation says.

User-agent: SemrushBot
Disallow: /

Is it really?

How to tell a real SemrushBot from an impostor

Anyone can put “SemrushBot” in a user agent. What can't be faked:

  • Its operator doesn't publish a way to verify it, so a request claiming to be it can only be treated as a claim.

In Punchline

How Punchline counts SemrushBot

Punchline labels a request that names SemrushBot with how sure it is: verified or declared when the evidence checks out, a lower tier when it can't be checked, and impostor when the evidence contradicts it. Each visit is counted under Semrush, with its purpose, so you can see whether SemrushBot is reading your site and which pages it reads. How it works · Known agents

See whether SemrushBot reads your site