AI Crawler Robots.txt Generator
Govern every major AI crawler from a single robots.txt. Pick a stance per bot: allow full access, block entirely, or allow only specific paths. Covers GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, Applebot-Extended, and more.
AI Visibility Intelligence
Pick a stance per crawler, then paste the generated rules into your site's robots.txt.
AI crawlers now split into three groups: training crawlers (GPTBot, ClaudeBot, Google-Extended) harvest content to train models, answer crawlers (PerplexityBot, ChatGPT-User) fetch pages when users ask a question, and indexing crawlers (OAI-SearchBot) build search indices. Blocking all three removes your brand from AI answers; allowing all three maximises AI visibility. Most enterprises land somewhere in the middle.
Quick-start presets
AI crawler controls
Per-bot configuration
Set a stance for each crawler. "Partial" lets you block specific paths further down.
Applies to AI crawlers that honour it. Not all do. Leave blank for none.
These paths will be disallowed for any bot you set to "Partial" below.
Policy summary
What your current config allows and blocks
Generated robots.txt
Paste into the root of your site
How to deploy
- Click Copy on the generated code above, or download as robots.txt.
- Open your existing robots.txt. On most sites it lives at
yourdomain.com/robots.txt. - Paste the AI crawler rules at the end of your file, or merge them in if you already have per-bot rules.
- Upload the updated file to your site root. On WordPress, use a plugin like Yoast or Rank Math (SEO and Tools and File editor). On Shopify, edit via
robots.txt.liquid. - Verify at
yourdomain.com/robots.txtin your browser. - Changes take effect on the next crawl of each bot. Enforcement varies: most reputable AI bots honour robots.txt; some ignore it.
Next steps
Track what AI engines say about you
SEORCE AI Beacon tracks your brand across 7 AI engines + Google AI Overview, AI Mode & Search, with the exact page behind each citation.
Frequently asked questions
What is an AI crawler?
An AI crawler is a bot operated by an AI company that visits sites to collect content for training large language models or for answering live user queries. The best-known are GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google Bard/Gemini training), PerplexityBot, Bytespider (ByteDance), and CCBot (Common Crawl).
Should I block AI crawlers?
That is a brand decision. Blocking stops your content from being used to train future models and from being cited in live AI answers. Allowing gets you visibility in AI engines but also lets your content train competing systems. Many publishers block training-only bots (GPTBot, Google-Extended) and allow live-answer bots (PerplexityBot).
How is this different from a normal robots.txt generator?
A normal generator covers Googlebot and Bingbot. This one adds the full list of AI crawler user-agents and gives you presets for common policies (allow all, block training, block everything).
Where do I put the generated robots.txt file?
At the root of your domain:
https://yourdomain.com/robots.txt. It must be served as plain text. Any CMS allows this through hosting, server config, or an SEO plugin.Does blocking GPTBot remove my content from ChatGPT today?
It stops future training on new content and stops ChatGPT from fetching your pages live for answers. Content already in the model stays until the next retraining cycle.
What is Google-Extended?
Google-Extended is Google's separate user-agent for Bard and Gemini training. Blocking Googlebot would hurt normal SEO. Blocking Google-Extended only stops the AI training feed.
How often do I need to update my robots.txt for AI crawlers?
Quarterly is reasonable. New AI companies launch crawlers regularly. The ones from the last 12 months include PerplexityBot, ClaudeBot variants, Meta-ExternalAgent, and Amazonbot.
Does blocking affect how my pages rank in Google Search?
No. Googlebot is a separate user-agent from Google-Extended, GPTBot, and the rest. Blocking AI bots does not touch normal search ranking.
What about rate-limiting instead of outright blocking?
Robots.txt itself does not support rate limiting beyond
Crawl-delay, which most AI crawlers ignore. Use Cloudflare or your WAF for proper rate limiting.Can I block AI crawlers at the firewall instead?
Yes, and that is more reliable because robots.txt is advisory. Cloudflare offers one-click AI bot blocking under Security and Bots. Combine both.
Is there an industry standard for AI crawler opt-out?
The emerging standard is
llms.txt and the IETF ai.txt draft, both still unofficial. Robots.txt remains the practical standard today.What is llms.txt?
A proposed file at
/llms.txt that gives AI engines a curated list of your most valuable content, essentially a sitemap optimised for LLM consumption. Supported by Anthropic and others.Free · no account
SEORCE for Chrome, Edge and Firefox
Audits the page you are on: headings, meta tags, schema, indexability, and whether AI crawlers can read it. Export the result as a PDF. Free, with no account.
It audits one page at a time and does not track AI answers. About the extension
Where you stand in AI answers
This tool checks the inputs. AI Beacon shows the answers.
SEORCE AI Beacon tracks whether your brand is named and cited across 7 AI engines + Google AI Overview, AI Mode & Search. It shows the exact page behind each citation, and whether it was cited or only consulted.
3,000+ users · Free forever plan, no card needed