How AI crawlers read your site: GPTBot, ClaudeBot, Google-Extended & friends

Who is actually crawling

Every major answer engine runs its own fetcher. OpenAI's GPTBot gathers training and retrieval data for ChatGPT. Anthropic's ClaudeBot does the same for Claude. Google-Extended is Google's switch for AI training use (separate from classic Googlebot search indexing), PerplexityBot feeds Perplexity's answer index, and CCBot builds Common Crawl — the public dataset many models train on.

The important nuance: blocking one of these does not block the others, and blocking a training crawler is not the same as blocking search. A site can rank #1 on Google while being invisible to every AI assistant — because someone blanket-blocked AI crawlers in robots.txt years ago and forgot.

robots.txt is the gate

Each crawler reads your robots.txt and honors the most specific User-agent group that names it, falling back to the wildcard (*) group otherwise. A 'Disallow: /' inside a crawler's group locks that engine out of your entire site. That single line is the most common reason a brand never appears in AI answers.

The fix is deliberate policy, not accident: decide which engines you want reading you, express exactly that, and declare your sitemap so whatever you allow gets discovered fast.

Check yourself, right now

Our free Robots.txt AI Crawler Checker reads your robots.txt exactly the way GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot do, and reports who is blocked — no signup. If you want the full picture (structured data, answer-readiness, llms.txt), run the AI Answer Authority Audit: 17 checks, severity-ranked, with the exact fixes.