AI agent directory · Google
Googlebot
Googlebot is Google's crawler for building the index its search and answers draw on. Here's how to recognize it and check a request really is it.
At a glance
Googlebot facts
- Operator: Google
- Type: Crawler
- Why it fetches: Builds a search or answer index
- User agent contains: `Googlebot`, `GoogleOther`, `Google-InspectionTool`
- Verified by: published IP ranges, reverse DNS
- Operator's documentation: developers.google.com/search/docs/crawling-indexing/verifying-googlebot
Is it really?
How to tell a real Googlebot from an impostor
Anyone can put "Googlebot" in a user agent. What can't be faked is below.
- Check the address. Google publishes the IP ranges Googlebot uses: developers.google.com/static/crawling/ipranges/common-crawlers.json. A request whose user agent says Googlebot from an address outside them is an impostor.
- Check reverse DNS. Look up the requesting IP's host name; it should end in `googlebot.com` or `google.com`. Then look that name up and make sure it resolves back to the same IP, since anyone can set a reverse DNS name on their own addresses.
In Punchline
How Punchline counts Googlebot
Punchline labels a request that names Googlebot with how sure it is: verified or declared when the evidence checks out, a lower tier when it can't be checked, and impostor when the evidence contradicts it. Each visit is counted under Google, with its purpose, so you can see whether Googlebot is reading your site and which pages it reads. How it works · Known agents