Free LLM SEO Checker — AI Crawler Access Audit
Paste your robots.txt and this free LLM SEO checker shows exactly which AI crawlers — ChatGPT, Claude, Perplexity, Gemini, Apple Intelligence — can actually read your site. No signup, nothing leaves your browser.
What This Checks
This tool parses your pasted robots.txt against the real, documented crawler names each major AI company publishes for its own bots — not a guess, and not the regular search crawler (Googlebot, Bingbot) you're probably already allowing. It tells you, bot by bot, whether your current rules allow, block, or simply don't mention each one — and what that default actually means.
The Crawlers Checked
| Crawler | Company | Purpose |
|---|---|---|
GPTBot | OpenAI | Model training |
OAI-SearchBot | OpenAI | ChatGPT search citations |
ChatGPT-User | OpenAI | Live user requests in ChatGPT |
ClaudeBot | Anthropic | Model training |
Claude-SearchBot | Anthropic | Search result quality |
Claude-User | Anthropic | Live user requests in Claude |
PerplexityBot | Perplexity | Indexing and crawling |
Perplexity-User | Perplexity | Live user requests |
Google-Extended | Gemini training and AI features grounding | |
Applebot-Extended | Apple | Apple Intelligence training |
CCBot | Common Crawl | Open dataset used by many AI labs |
Why "Not Mentioned" Isn't the Same as "Blocked"
Robots.txt works on explicit rules. If a crawler isn't named anywhere in the file, it falls back to whatever the wildcard User-agent: * block allows — and if there's no wildcard block either, the default is to allow everything. A site that never updated its robots.txt for AI crawlers is almost always fully open to all of them by default, not blocked, which surprises most people checking this for the first time.
Google-Extended Is Different From the Rest
Google-Extended isn't a crawler that visits your site — it's a control token read by the regular Googlebot. Blocking it doesn't stop Googlebot from crawling and indexing your pages normally; it only opts that already-crawled content out of being used to train Gemini or ground Google's AI features. Every other bot on this list is a separate crawler with its own distinct visits and its own IP ranges.
LLM SEO Checker FAQs
Does blocking these bots remove my site from ChatGPT or Claude's existing answers?
No. Blocking a crawler stops future crawling, but it doesn't retroactively remove content a model may have already been trained on in a prior crawl.
If I block GPTBot, am I also blocking ChatGPT-User?
No — each company runs separate bots for training versus live user requests versus search indexing, and blocking one does not block the others. Each needs its own rule if you want it blocked specifically.
Does ChatGPT-User or Claude-User respect robots.txt at all?
These user-triggered bots generally fetch a specific page because a person asked a direct question, and the companies' own documentation notes robots.txt may not apply the same way it does to autonomous crawling bots like GPTBot or ClaudeBot.
Is there a downside to blocking all AI crawlers?
Blocking training bots (GPTBot, ClaudeBot) has no visibility downside since they don't power live answers. Blocking search/citation bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) does mean your pages won't be cited or linked in those tools' answers.
Does this tool fetch my live robots.txt for me?
No — paste the contents directly. Fetching an arbitrary external URL from inside your browser is blocked by cross-origin browser security for most sites, so pasting keeps the check reliable and keeps your data off any server, including ours.
Does this tool store or send my data anywhere?
No. Everything is checked in your browser with JavaScript — nothing is uploaded, logged, or stored on any server, including ours.
Want a full AI search visibility strategy?
Contomatix handles AI SEO end to end, not just crawler access rules.