AI agent directory · Common Crawl
CCBot
CCBot is Common Crawl's crawler for collecting pages that may be used to train AI models. Here's how to recognize it and check a request really is it.
At a glance
CCBot facts
- Operator: Common Crawl
- Type: Crawler
- Why it fetches: Collects training data
- User agent contains: `CCBot`
- Verified by: published IP ranges
- Operator's documentation: commoncrawl.org/ccbot
Is it really?
How to tell a real CCBot from an impostor
Anyone can put "CCBot" in a user agent. What can't be faked is below.
- Check the address. Common Crawl publishes the IP ranges CCBot uses: index.commoncrawl.org/ccbot.json. A request whose user agent says CCBot from an address outside them is an impostor.
In Punchline
How Punchline counts CCBot
Punchline labels a request that names CCBot with how sure it is: verified or declared when the evidence checks out, a lower tier when it can't be checked, and impostor when the evidence contradicts it. Each visit is counted under Common Crawl, with its purpose, so you can see whether CCBot is reading your site and which pages it reads. How it works · Known agents