Free Robots.txt Tester — Check If a URL Is Allowed or Blocked
Paste your robots.txt, choose a crawler and list the URLs you want to check. This free robots.txt tester applies Google's matching rules, shows which rule decided each URL and flags lines Google ignores. No signup, nothing leaves your browser.
Paste a robots.txt and add URLs to test.
How the Tester Decides
- One group applies. Google uses the group whose user agent is the most specific match for the crawler. Rules from groups for the same agent are combined, and the
*group is used only when no specific group matches. - The longest matching rule wins. Each Allow or Disallow path is compared with the start of the URL path, and the most specific rule, judged by path length, decides.
- A tie goes to Allow. When an Allow and a Disallow match with the same length, Google uses the least restrictive rule.
- Wildcards.
*matches any run of characters and$marks the end of the URL. - Paths are case-sensitive.
/Adminand/adminare different rules. - Empty Disallow blocks nothing. A rule with no path is ignored.
What a Robots.txt Test Can and Cannot Tell You
The result tells you whether a crawler that follows these rules may fetch a URL. It does not tell you whether the page is indexed. A URL blocked by robots.txt can still appear in search results if other pages link to it, and a page you allow may still be left out of the index for other reasons. If the page must stay out of search, a noindex rule on a crawlable page is the right tool.
The tool follows the rules Google documents. Other crawlers can differ in small ways, so treat a result for a non-Google crawler as a close guide, and check that crawler's own documentation if the outcome matters. Google also supports only the user-agent, allow, disallow and sitemap fields and ignores others, such as crawl-delay, and it enforces a size limit of 500 KiB on the file.
Robots.txt Tester FAQs
Does the tester fetch my live robots.txt?
No. Paste the contents. Browsers block most cross-site requests to other people's files, and pasting keeps your data off every server, including ours.
Which rule wins when Allow and Disallow both match?
The longer, more specific path wins. If both are the same length, Google uses the least restrictive rule, which is Allow.
Why does my URL show as allowed when I have a Disallow for its folder?
A longer Allow rule may be overriding it, or the rule may sit in a group for a different crawler. The result lists the rule that decided each URL so you can see which one it was.
Can I test a crawler like GPTBot or ClaudeBot?
Yes. Type the crawler's name in the crawler field. If your file has no group for it, the tester falls back to the * group, as Google does.
Does this tool store or send my data anywhere?
No. The rules are applied in your browser with JavaScript — nothing is uploaded, logged, or stored on any server, including ours.
Not sure what your robots.txt should block?
Contomatix reviews crawl rules as part of a technical SEO check.