Robots.txt AI Crawler Checker
If an AI engine’s crawler can’t read your site, that engine can’t cite you. This checker reads your robots.txt the way GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot do and tells you exactly who is locked out.
Want the full picture — structured data, answer-readiness, llms.txt? Run the free AI Answer Authority Audit.
Asked before checking
Can ChatGPT crawl my website?
Only if your robots.txt allows OpenAI's agents. GPTBot is the training crawler; OAI-SearchBot builds the index ChatGPT's answers cite from; ChatGPT-User fetches pages live for a user. Blocking the last two removes you from ChatGPT answers — run the check above to see your current verdict.
Should I block AI crawlers?
Split the decision by crawler type. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) keeps your content out of future model training without affecting today's citations. Blocking search-index or user-triggered agents (OAI-SearchBot, PerplexityBot, Claude-User) removes you from AI answers entirely. Many sites blanket-blocked everything in 2023 and are invisible to AI engines today without knowing it.
Which crawlers does this checker test?
GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot — parsed from your robots.txt exactly the way the crawlers parse it (group matching, wildcards, most-specific-wins). It also checks whether your robots.txt declares a sitemap.
Engine-by-engine playbooks — including which agents to allow and why: the GEO guides →