GPTBot, ClaudeBot, PerplexityBot: which AI crawlers to let in

The short answer: most AI companies now split their crawlers into ones that collect training data and ones that fetch pages to answer a user's question right now. If you want to show up in AI answers, allow the search and user crawlers. Whether you allow training crawlers is a separate choice.

Why there are so many crawlers

A single "AI bot" would force an all-or-nothing choice. So the big providers publish several user agents, each for one job. Blocking the wrong one can quietly remove you from AI answers.

The main ones

OpenAI

• GPTBot: collects data that may be used to train models
• OAI-SearchBot: finds pages for ChatGPT's search results
• ChatGPT-User: fetches a page when a user's request needs it

Anthropic

• ClaudeBot: collects data that may be used for training
• Claude-SearchBot: indexes pages to improve search results in Claude
• Claude-User: fetches a page when a user asks Claude to

Perplexity

• PerplexityBot: indexes pages for Perplexity's answers
• Perplexity-User: fetches a page for a user's question

Google

• Googlebot: Google Search, including its AI features
• Google-Extended: a separate token that controls whether your content is used for Gemini training. It doesn't affect Google Search.

Providers update these lists from time to time, so check their documentation before you rely on any rule.

A sensible default for a product site

If you want to be found and quoted, but would rather not be used for training, this is a reasonable start:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /
Disallow: /api/
Disallow: /dashboard

Search and user crawlers fall under the last group and can read your public pages. Private areas stay off limits for everyone.

Robots.txt is a request, not a lock

Well-behaved crawlers follow it. It doesn't protect anything private. Pages that shouldn't be public need a login, not a robots rule.

The takeaway

Decide crawler by crawler. Let in the bots that put you in answers, make a deliberate choice about training, and keep private pages behind a login.