Robots.txt AI Crawler Checker
If an AI engine’s crawler can’t read your site, that engine can’t cite you. This checker reads your robots.txt the way the major training, answer-index and user-fetch agents do and tells you exactly who is locked out — and whether it costs you citations or just training opt-out.
Want the full picture — structured data, answer-readiness, llms.txt? Run the free AI Answer Authority Audit.
Asked before checking
Can ChatGPT crawl my website?
Only if your robots.txt allows OpenAI's agents. GPTBot is the training crawler; OAI-SearchBot builds the index ChatGPT's answers cite from; ChatGPT-User fetches pages live for a user. Blocking the last two removes you from ChatGPT answers — run the check above to see your current verdict.
Should I block AI crawlers?
Split the decision by crawler type. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) keeps your content out of future model training without affecting today's citations. Blocking search-index or user-triggered agents (OAI-SearchBot, PerplexityBot, Claude-User) removes you from AI answers entirely. Many sites blanket-blocked everything in 2023 and are invisible to AI engines today without knowing it.
Which crawlers does this checker test?
The major AI agents across all three roles — training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended), answer-index bots (OAI-SearchBot, PerplexityBot, Amazonbot), and user-fetch agents (ChatGPT-User, Claude-User, Perplexity-User) — parsed from your robots.txt exactly the way the crawlers parse it (group matching, wildcards, most-specific-wins, Allow/Disallow precedence). It also checks whether your robots.txt declares a sitemap.
Engine-by-engine playbooks — including which agents to allow and why: the GEO guides →