Free Robots.txt Generator
This free robots.txt generator builds a valid robots.txt file in seconds — control which crawlers can access your site and point them to your sitemap. No signup, nothing leaves your browser.
User-agent: * Allow: /
Upload this to your site's root directory so it's reachable at yoursite.com/robots.txt.
What Is Robots.txt?
robots.txt is a plain-text file at your domain's root that tells search engine and AI crawlers which parts of your site they're allowed to request. It's the first file most well-behaved crawlers check before visiting anything else. It doesn't hide pages from people who already have the link, and it isn't a security tool — it's a politeness convention that reputable crawlers (Googlebot, Bingbot) respect.
Robots.txt vs. Meta Robots vs. noindex
| Method | Controls | Common mistake |
|---|---|---|
| robots.txt | Whether a crawler requests the page at all | Blocking a page here does NOT remove it from Google if it's already indexed and linked elsewhere |
Meta noindex tag | Whether a crawled page gets indexed | If robots.txt blocks the page, Google can't see the noindex tag at all — the two can work against each other |
To actually remove a page from search results, let crawlers access it (don't block it in robots.txt) but add a <meta name="robots" content="noindex"> tag to the page itself.
Should You Block AI Crawlers?
This is a genuine judgment call, not a technical necessity. Blocking bots like GPTBot (OpenAI) or CCBot (Common Crawl) stops your content from being used to train future AI models, but it may also reduce your chances of being cited by AI answer engines that rely on the same or related crawl data. There's no universally "correct" answer — publishers who want maximum AI visibility typically leave these open; publishers concerned about content scraping block them.
Common Robots.txt Mistakes
- Blocking your entire site by accident — a stray
Disallow: /left over from a staging environment is one of the most common self-inflicted SEO disasters - Forgetting the sitemap reference — it costs one line and helps crawlers discover your pages faster
- Trying to block a page that's already indexed — use a noindex tag instead, or the page can persist in search results with no description
- Putting the file anywhere but the root —
yoursite.com/robots.txtonly, a subfolder location is ignored entirely
Free Robots.txt Generator FAQs
Does robots.txt guarantee a page won't appear in Google?
No. If other sites link to a blocked page, Google can still index the URL itself (usually with no description, since it couldn't crawl the content). Use a noindex meta tag for pages you genuinely want removed from search results.
Will blocking AI crawlers hurt my SEO?
Not for traditional Google/Bing search rankings — those use separate crawlers (Googlebot, Bingbot) that this doesn't affect. It may reduce visibility in AI-generated answers that rely on the blocked crawlers' data.
Do I need a robots.txt file at all?
Not strictly — crawlers will assume everything is allowed if no file exists. Having one is still good practice: it lets you reference your sitemap and gives you control if you ever need to restrict something.
Can I have different rules for different crawlers?
Yes — robots.txt supports separate User-agent blocks for specific bots. This tool generates a simple universal ruleset; for per-bot rules, edit the downloaded file directly.
Does this tool store or send my data anywhere?
No. Everything is generated in your browser with JavaScript — nothing is uploaded, logged, or stored on any server, including ours.
Want your whole technical SEO handled?
Contomatix audits crawlability, indexing, and site architecture, not just single files.