AI Crawlers — GPTBot, ClaudeBot, PerplexityBot and robots.txt
TL;DR
Bots run by AI companies that read your pages for model training, for an AI search index, or when a user asks. Training and search bots are separate tokens in robots.txt, so you can be cited in AI answers without joining training data.
Explain Simply
Think of three visitors from the same company: one copies books for a library that teaches students (training), one writes the catalog card so people can find your book (search index), and one fetches the book because a reader asked for it right now (user fetch). robots.txt lets you say yes to some and no to others.
AI crawlers are the bots AI vendors send to the web. Each vendor separates them by job:
Training: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl). Google-Extended and Applebot-Extended are control tokens only: they do not crawl, they decide whether content Googlebot or Applebot already fetched may train Gemini or Apple's models.
Search index: OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot. Blocking these keeps you out of that engine's AI answers.
User-initiated fetch: ChatGPT-User, Claude-User, Perplexity-User fetch one page because a user asked. OpenAI says robots.txt may not apply to ChatGPT-User; Perplexity says Perplexity-User generally ignores it.
robots.txt grouping (RFC 9309): a crawler follows only the most specific group that matches it. A User-agent: GPTBot group with Allow: / replaces the * group for GPTBot, so the Disallow lines under * stop applying to it.
Most AI crawlers do not run JavaScript: content that only appears after JS runs is invisible to them.
⚡Why It Matters
Blocking the wrong token is the most common mistake: blocking GPTBot does not remove a site from ChatGPT search, and Google-Extended does not affect Google Search inclusion or ranking. Blocking a search bot by accident, or opening private folders through a grouping mistake, costs visibility or exposes pages you meant to keep out.
🔍How CiteStat Checks
CiteStat fetches robots.txt and reads it with RFC 9309 group matching for 10 AI tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, CCBot, Bytespider, Google-Extended, cohere-ai). It lists each crawler kept off the site root, and a rule checks whether robots.txt names AI crawlers explicitly.
CiteStat Score Impact
GEO / AI Visibility card: −1 point per blocked AI crawler, at most −3 of 10. Rules engine: "AI crawlers should not be blocked in robots.txt" (info, weight 3) and "robots.txt should explicitly address AI crawlers" (info, weight 3).
Fix Difficulty
Editor: Easy | Developer: Easy | AI Agent: Easy
Guides on this topic
- Web Bot Auth: How AI Agents Prove Who They Are
- Content Signals in robots.txt: search, ai-input, ai-train and the Wrong-Group Mistake
- AI Overviews and nosnippet, max-snippet: How to Control What Google's AI Quotes
- Markdown for AI Agents: What Accept: text/markdown Does and Whether to Turn It On
- Do AI Crawlers Run JavaScript? What GPTBot and ClaudeBot See
- AI Crawlers and robots.txt: Should You Block GPTBot and ClaudeBot?