# Should you block AI crawlers?

> "AI crawlers" are at least three different things, training, search and live answers, and blocking each one costs something different. A decision guide with the exact robots.txt names.

Punchline Analytics · 2026-11-12

**Short answer:** decide per purpose, not per company. Blocking **training** crawlers keeps your content out of future models and costs you little in visibility. Blocking **search** crawlers takes you out of AI search answers. Blocking **live fetchers** stops assistants reading your page when someone asks about it, and they may not follow robots.txt anyway. Most businesses that want to be found should allow search and live fetches, and make a deliberate choice about training.

## Three purposes, three trade-offs

Each major AI company runs separate crawlers for separate jobs, and documents each one's name for robots.txt:

| Purpose | OpenAI | Anthropic | Perplexity | Google |
|---|---|---|---|---|
| Training data | `GPTBot` | `ClaudeBot` | none | `Google-Extended` (a control token, not a crawler) |
| Search and answer index | `OAI-SearchBot` | `Claude-SearchBot` | `PerplexityBot` | `Googlebot` |
| Live fetch for a person asking | `ChatGPT-User` | `Claude-User` | `Perplexity-User` | |

Sources: [OpenAI](https://developers.openai.com/api/docs/bots), [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers), [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers).

### Training crawlers

OpenAI says disallowing `GPTBot` signals that your content shouldn't be used to train its foundation models, and that this is independent of its search crawler. Anthropic says the same of `ClaudeBot`. Google's `Google-Extended` token controls whether content Google crawls can be used to train future Gemini models, and Google says it [doesn't affect inclusion or ranking in Google Search](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers).

**Block if** your content is your product (publishers, paid research, proprietary docs), or you simply don't want it in training data. **The cost** is hard to measure and probably small for visibility today, since search and live answers use the other crawlers.

### Search crawlers

`OAI-SearchBot` decides whether you appear in ChatGPT's search answers; OpenAI says blocking it hides your site from them. Anthropic says blocking `Claude-SearchBot` can reduce your visibility in Claude's search results. Perplexity recommends allowing `PerplexityBot`, and says it doesn't use it to train foundation models.

**Block if** you'd rather not appear in AI answers at all. **The cost** is real: assistants can't recommend or link to what they can't find.

### Live fetchers

When a person asks an assistant something that needs your page, `ChatGPT-User`, `Claude-User` or `Perplexity-User` fetches it on the spot. Anthropic says blocking `Claude-User` stops retrieval for those questions. OpenAI says robots.txt rules [may not apply](https://developers.openai.com/api/docs/bots) to `ChatGPT-User`, since a person asked, and Perplexity says `Perplexity-User` [generally ignores robots.txt](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) for the same reason.

**Block if** a page truly shouldn't be read by anyone's assistant. **The cost** is answering "what does this company charge?" with someone else's page.

## A sensible default

For a business that wants to be found:

```
# Allow AI search and live answers (the defaults; no rule needed).
# Opt out of model training:
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /
```

Then keep the parts nobody needs out for everyone, for example `Disallow: /account/` under `User-agent: *`.

Two caveats. Robots.txt is a request, not a lock: well-run crawlers follow it, impostors don't, and changes can take about a day to apply ([OpenAI](https://developers.openai.com/api/docs/bots) and [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) both say so). And a crawler is only who it says it is if its address checks out; see [how to verify one](https://punchlineanalytics.com/agents).

## Decide with data

Before blocking anything, look at what each crawler actually does on your site: which pages it reads, how often, and whether the assistants behind it send you people. Punchline shows that per crawler and purpose, and flags impostors using the big names. [See it on your own site](https://app.punchlineanalytics.com/signup).
