punchline!

Blog · November 12, 2026 · Punchline Analytics

Should you block AI crawlers?

"AI crawlers" are at least three different things, training, search and live answers, and blocking each one costs something different. A decision guide with the exact robots.txt names.

Short answer: decide per purpose, not per company. Blocking training crawlers keeps your content out of future models and costs you little in visibility. Blocking search crawlers takes you out of AI search answers. Blocking live fetchers stops assistants reading your page when someone asks about it, and they may not follow robots.txt anyway. Most businesses that want to be found should allow search and live fetches, and make a deliberate choice about training.

Three purposes, three trade-offs

Each major AI company runs separate crawlers for separate jobs, and documents each one's name for robots.txt:

Purpose OpenAI Anthropic Perplexity Google
Training data GPTBot ClaudeBot none Google-Extended (a control token, not a crawler)
Search and answer index OAI-SearchBot Claude-SearchBot PerplexityBot Googlebot
Live fetch for a person asking ChatGPT-User Claude-User Perplexity-User

Sources: OpenAI, Anthropic, Perplexity, Google.

Training crawlers

OpenAI says disallowing GPTBot signals that your content shouldn't be used to train its foundation models, and that this is independent of its search crawler. Anthropic says the same of ClaudeBot. Google's Google-Extended token controls whether content Google crawls can be used to train future Gemini models, and Google says it doesn't affect inclusion or ranking in Google Search.

Block if your content is your product (publishers, paid research, proprietary docs), or you simply don't want it in training data. The cost is hard to measure and probably small for visibility today, since search and live answers use the other crawlers.

Search crawlers

OAI-SearchBot decides whether you appear in ChatGPT's search answers; OpenAI says blocking it hides your site from them. Anthropic says blocking Claude-SearchBot can reduce your visibility in Claude's search results. Perplexity recommends allowing PerplexityBot, and says it doesn't use it to train foundation models.

Block if you'd rather not appear in AI answers at all. The cost is real: assistants can't recommend or link to what they can't find.

Live fetchers

When a person asks an assistant something that needs your page, ChatGPT-User, Claude-User or Perplexity-User fetches it on the spot. Anthropic says blocking Claude-User stops retrieval for those questions. OpenAI says robots.txt rules may not apply to ChatGPT-User, since a person asked, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason.

Block if a page truly shouldn't be read by anyone's assistant. The cost is answering "what does this company charge?" with someone else's page.

A sensible default

For a business that wants to be found:

# Allow AI search and live answers (the defaults; no rule needed).
# Opt out of model training:
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Then keep the parts nobody needs out for everyone, for example Disallow: /account/ under User-agent: *.

Two caveats. Robots.txt is a request, not a lock: well-run crawlers follow it, impostors don't, and changes can take about a day to apply (OpenAI and Perplexity both say so). And a crawler is only who it says it is if its address checks out; see how to verify one.

Decide with data

Before blocking anything, look at what each crawler actually does on your site: which pages it reads, how often, and whether the assistants behind it send you people. Punchline shows that per crawler and purpose, and flags impostors using the big names. See it on your own site.

Start free