Free Tool

Free LLM SEO Checker: AI Crawler Access Audit

Paste your robots.txt and this free LLM SEO checker shows exactly which AI crawlers. ChatGPT, Claude, Perplexity, Gemini, Apple Intelligence, can actually read your site. No signup, nothing leaves your browser.

AI crawler access Waiting for input
Paste a robots.txt file to check it.
How it works

Run an AI crawler check in three steps

01

Copy your robots.txt

Open yourdomain.com/robots.txt in a browser and copy the whole file.

02

Paste it in

Paste it into the box. The tool does not fetch your file for you, and nothing leaves your browser.

03

Read the verdicts

Each of the 11 AI crawlers is marked allowed or blocked, with a count such as "9 of 11 AI crawlers allowed".

What This Checks

This tool parses your pasted robots.txt against the real, documented crawler names each major AI company publishes for its own bots, not a guess, and not the regular search crawler (Googlebot, Bingbot) you're probably already allowing. It tells you, bot by bot, whether your current rules allow, block, or simply don't mention each one, and what that default actually means.

The Crawlers Checked

CrawlerCompanyPurpose
GPTBotOpenAIModel training
OAI-SearchBotOpenAIChatGPT search citations
ChatGPT-UserOpenAILive user requests in ChatGPT
ClaudeBotAnthropicModel training
Claude-SearchBotAnthropicSearch result quality
Claude-UserAnthropicLive user requests in Claude
PerplexityBotPerplexityIndexing and crawling
Perplexity-UserPerplexityLive user requests
Google-ExtendedGoogleGemini training and AI features grounding
Applebot-ExtendedAppleApple Intelligence training
CCBotCommon CrawlOpen dataset used by many AI labs

Why "Not Mentioned" Isn't the Same as "Blocked"

Robots.txt works on explicit rules. If a crawler isn't named anywhere in the file, it falls back to whatever the wildcard User-agent: * block allows, and if there's no wildcard block either, the default is to allow everything. A site that never updated its robots.txt for AI crawlers is almost always fully open to all of them by default, not blocked, which surprises most people checking this for the first time.

Google-Extended Is Different From the Rest

Google-Extended isn't a crawler that visits your site, it's a control token read by the regular Googlebot. Blocking it doesn't stop Googlebot from crawling and indexing your pages normally; it only opts that already-crawled content out of being used to train Gemini or ground Google's AI features. Every other bot on this list is a separate crawler with its own distinct visits and its own IP ranges.

Related Questions About LLM SEO Checkers

LLM SEO Checker: What It Checks

This LLM SEO checker checks one thing precisely: whether your robots.txt lets the main AI crawlers read your site. It covers 11 of them, from OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl, and shows which are allowed and which are blocked.

LLM SEO Check: What a Robots.txt Check Cannot Tell You

Access is the first condition for being cited, not a guarantee of it. A robots.txt check cannot tell you whether an AI assistant mentions your brand, or how your pages are written. For that side of the work, read what is LLM SEO and how to get cited by ChatGPT.

LLM SEO Checking Tool: How to Run the Check

Copy your robots.txt from yourdomain.com/robots.txt and paste it into the box. The report lists every crawler with its company and purpose, such as model training, search citations or live user requests, so you can see which parts of your access are open and which are closed.

Why One AI Crawler Can Be Blocked While Another Is Not

The crawlers do different jobs, so a site can allow search citations and still block training. Blocking GPTBot is not the same as blocking ChatGPT-User, and Google-Extended works differently from the rest, as the sections above explain.

LLM SEO Analysis Tool: Where This Fits in Your Workflow

Use this check after you edit robots.txt, and test individual URLs with the robots.txt tester. If you want to give AI assistants a plain-language map of your site, build one with the llms.txt generator.

LLM SEO Checker FAQs

Does blocking these bots remove my site from ChatGPT or Claude's existing answers?

No. Blocking a crawler stops future crawling, but it doesn't retroactively remove content a model may have already been trained on in a prior crawl.

If I block GPTBot, am I also blocking ChatGPT-User?

No, each company runs separate bots for training versus live user requests versus search indexing, and blocking one does not block the others. Each needs its own rule if you want it blocked specifically.

Does ChatGPT-User or Claude-User respect robots.txt at all?

These user-triggered bots generally fetch a specific page because a person asked a direct question, and the companies' own documentation notes robots.txt may not apply the same way it does to autonomous crawling bots like GPTBot or ClaudeBot.

Is there a downside to blocking all AI crawlers?

Blocking training bots (GPTBot, ClaudeBot) has no visibility downside since they don't power live answers. Blocking search/citation bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) does mean your pages won't be cited or linked in those tools' answers.

Does this tool fetch my live robots.txt for me?

No, paste the contents directly. Fetching an arbitrary external URL from inside your browser is blocked by cross-origin browser security for most sites, so pasting keeps the check reliable and keeps your data off any server, including ours.

Does this tool store or send my data anywhere?

No. Everything is checked in your browser with JavaScript, nothing is uploaded, logged, or stored on any server, including ours.

Is there a free LLM SEO checker?

Yes. This one is free, needs no signup and runs in your browser. You paste your robots.txt and it shows which AI crawlers can read your site.

Does an LLM SEO check tell me if ChatGPT cites my site?

No. It checks whether your robots.txt allows AI crawlers to read your pages, which is only the first step. Whether you are cited depends on your content and your authority.

Keep Going: Related Tools and Guides

Want a full AI search visibility strategy?

Contomatix handles AI SEO end to end, not just crawler access rules.