Free LLM SEO Checker: AI Crawler Access Audit
Paste your robots.txt and this free LLM SEO checker shows exactly which AI crawlers. ChatGPT, Claude, Perplexity, Gemini, Apple Intelligence, can actually read your site. No signup, nothing leaves your browser.
Run an AI crawler check in three steps
Copy your robots.txt
Open yourdomain.com/robots.txt in a browser and copy the whole file.
Paste it in
Paste it into the box. The tool does not fetch your file for you, and nothing leaves your browser.
Read the verdicts
Each of the 11 AI crawlers is marked allowed or blocked, with a count such as "9 of 11 AI crawlers allowed".
What This Checks
This tool parses your pasted robots.txt against the real, documented crawler names each major AI company publishes for its own bots, not a guess, and not the regular search crawler (Googlebot, Bingbot) you're probably already allowing. It tells you, bot by bot, whether your current rules allow, block, or simply don't mention each one, and what that default actually means.
The Crawlers Checked
| Crawler | Company | Purpose |
|---|---|---|
GPTBot | OpenAI | Model training |
OAI-SearchBot | OpenAI | ChatGPT search citations |
ChatGPT-User | OpenAI | Live user requests in ChatGPT |
ClaudeBot | Anthropic | Model training |
Claude-SearchBot | Anthropic | Search result quality |
Claude-User | Anthropic | Live user requests in Claude |
PerplexityBot | Perplexity | Indexing and crawling |
Perplexity-User | Perplexity | Live user requests |
Google-Extended | Gemini training and AI features grounding | |
Applebot-Extended | Apple | Apple Intelligence training |
CCBot | Common Crawl | Open dataset used by many AI labs |
Why "Not Mentioned" Isn't the Same as "Blocked"
Robots.txt works on explicit rules. If a crawler isn't named anywhere in the file, it falls back to whatever the wildcard User-agent: * block allows, and if there's no wildcard block either, the default is to allow everything. A site that never updated its robots.txt for AI crawlers is almost always fully open to all of them by default, not blocked, which surprises most people checking this for the first time.
Google-Extended Is Different From the Rest
Google-Extended isn't a crawler that visits your site, it's a control token read by the regular Googlebot. Blocking it doesn't stop Googlebot from crawling and indexing your pages normally; it only opts that already-crawled content out of being used to train Gemini or ground Google's AI features. Every other bot on this list is a separate crawler with its own distinct visits and its own IP ranges.
Related Questions About LLM SEO Checkers
LLM SEO Checker: What It Checks
This LLM SEO checker checks one thing precisely: whether your robots.txt lets the main AI crawlers read your site. It covers 11 of them, from OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl, and shows which are allowed and which are blocked.
LLM SEO Check: What a Robots.txt Check Cannot Tell You
Access is the first condition for being cited, not a guarantee of it. A robots.txt check cannot tell you whether an AI assistant mentions your brand, or how your pages are written. For that side of the work, read what is LLM SEO and how to get cited by ChatGPT.
LLM SEO Checking Tool: How to Run the Check
Copy your robots.txt from yourdomain.com/robots.txt and paste it into the box. The report lists every crawler with its company and purpose, such as model training, search citations or live user requests, so you can see which parts of your access are open and which are closed.
Why One AI Crawler Can Be Blocked While Another Is Not
The crawlers do different jobs, so a site can allow search citations and still block training. Blocking GPTBot is not the same as blocking ChatGPT-User, and Google-Extended works differently from the rest, as the sections above explain.
LLM SEO Analysis Tool: Where This Fits in Your Workflow
Use this check after you edit robots.txt, and test individual URLs with the robots.txt tester. If you want to give AI assistants a plain-language map of your site, build one with the llms.txt generator.
LLM SEO Checker FAQs
Does blocking these bots remove my site from ChatGPT or Claude's existing answers?
No. Blocking a crawler stops future crawling, but it doesn't retroactively remove content a model may have already been trained on in a prior crawl.
If I block GPTBot, am I also blocking ChatGPT-User?
No, each company runs separate bots for training versus live user requests versus search indexing, and blocking one does not block the others. Each needs its own rule if you want it blocked specifically.
Does ChatGPT-User or Claude-User respect robots.txt at all?
These user-triggered bots generally fetch a specific page because a person asked a direct question, and the companies' own documentation notes robots.txt may not apply the same way it does to autonomous crawling bots like GPTBot or ClaudeBot.
Is there a downside to blocking all AI crawlers?
Blocking training bots (GPTBot, ClaudeBot) has no visibility downside since they don't power live answers. Blocking search/citation bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) does mean your pages won't be cited or linked in those tools' answers.
Does this tool fetch my live robots.txt for me?
No, paste the contents directly. Fetching an arbitrary external URL from inside your browser is blocked by cross-origin browser security for most sites, so pasting keeps the check reliable and keeps your data off any server, including ours.
Does this tool store or send my data anywhere?
No. Everything is checked in your browser with JavaScript, nothing is uploaded, logged, or stored on any server, including ours.
Is there a free LLM SEO checker?
Yes. This one is free, needs no signup and runs in your browser. You paste your robots.txt and it shows which AI crawlers can read your site.
Does an LLM SEO check tell me if ChatGPT cites my site?
No. It checks whether your robots.txt allows AI crawlers to read your pages, which is only the first step. Whether you are cited depends on your content and your authority.
Keep Going: Related Tools and Guides
Want a full AI search visibility strategy?
Contomatix handles AI SEO end to end, not just crawler access rules.