Robots.txt Generator
Build a complete robots.txt with enterprise-ready presets for Shopify, WordPress, Magento, headless commerce, and SaaS. Includes AI crawler controls and sitemap references. For AI-crawler-only controls, use the dedicated AI Crawler Robots.txt Generator.
Technical SEO Intelligence
A broken robots.txt blocks pages Google was ranking. Treat it as production code.
The three most expensive robots.txt mistakes: disallowing / (nukes your site from search), blocking faceted navigation too broadly (removes category pages), and referencing wrong sitemap URLs. This tool starts from a vetted preset and lets you edit without introducing those errors.
Platform preset
User-agent groups
Sitemaps & global directives
Generated robots.txt
Output
Next steps
Verify your robots.txt doesn't block pages in your sitemap
The Robots+Sitemap Conflict Checker finds URLs you want indexed but have accidentally blocked.
Frequently asked questions
What is robots.txt?
A text file at
/robots.txt that tells crawlers which parts of your site to crawl or not crawl. First file bots fetch.Does Disallow stop pages being indexed?
No. Disallow stops crawling. Google can still index URLs it finds elsewhere and show them in SERPs without a snippet. For true "don't index" use meta robots noindex.
What is User-agent: *?
A rule block that applies to all bots. Specific user-agents (Googlebot, Bingbot) can have their own rule blocks that override.
Should I disallow /admin and /wp-login.php?
They're already behind authentication so there's nothing to index. Disallowing is cosmetic security but doesn't hurt.
How do I handle sitemaps in robots.txt?
Add
Sitemap: https://example.com/sitemap.xml at the top. One line per sitemap. Full URL required.What about Crawl-delay?
A directive to space out crawler requests. Googlebot ignores it. Bing honors it. Set it via Search Console per-crawler instead.
What is noindex in robots.txt?
Google supported it unofficially until 2019 and now ignores it. Use meta robots noindex or X-Robots-Tag header instead.
How large can robots.txt be?
Google processes up to 500KB. Enterprise sites rarely exceed 10KB.
Does case matter in robots.txt?
Directives (User-agent, Disallow) are case-insensitive. Paths are case-sensitive on most servers.
What happens if robots.txt returns 404?
Google treats it as "crawl everything". Benign but you lose the ability to control crawl.
How do I test my robots.txt?
Google Search Console and Legacy tools and robots.txt Tester shows how Googlebot interprets each rule.
Free · no account
SEORCE for Chrome, Edge and Firefox
Audits the page you are on: headings, meta tags, schema, indexability, and whether AI crawlers can read it. Export the result as a PDF. Free, with no account.
It audits one page at a time and does not track AI answers. About the extension
Across the whole site
One check here. Every page in SEORCE.
This tool does one job. SEORCE Technical Suite crawls your whole site with fast and secure crawlers and ranks technical issues by impact. AutoFix applies fixes for meta tags, schema, canonical tags, image alt text, headings and internal links for orphan pages on your live site, through one script tag, and the list keeps growing.
3,000+ users · Free forever plan, no card needed