GUIDE · PART OF AI CRAWLER OPTIMIZATION
robots.txt for AI crawlers:
one file, a dozen bots.
robots.txt was written for search engines, and it now has to speak to a dozen AI bots as well. The syntax hasn't changed. What catches people out is how the rules combine, and which bots take any notice. This guide covers both, with examples you can adapt.
How does robots.txt work for AI crawlers?
A robots.txt file tells crawlers which parts of a site they may fetch. For AI crawlers it works as it does for search engines: each bot looks for a group of rules addressed to its own name, and uses the general group only if it finds none. AI companies run several bots each, for training, for search and for fetching a page on a person's behalf, and each bot has its own name to address.
Key facts
- Each bot follows the group that names it. Only bots with no group of their own use the general one
- A bot with its own group ignores the general rules completely, so shared rules have to be repeated
- Training bots, search bots and user-triggered fetchers have different names, and need different rules
- robots.txt is a request. Reputable crawlers honour it, and it does nothing to stop one that chooses to ignore it
- A firewall or CDN rule acts before robots.txt is read, and can block a bot you meant to allow
- Google-Extended is a token with no crawler of its own. It controls how Google uses your content for Gemini
Last reviewed 15 October 2026 by the SEORCE team.
The syntax is easy. The way rules combine isn't.
Almost every robots.txt mistake with AI bots comes from one of three misunderstandings. None of them is about how to write a rule.
A named bot reads one group
Give GPTBot its own group and it stops following your general rules. Anything you wanted it to obey has to be in that group too.
New bots keep appearing
A file written last year won't mention bots launched since. Names change as well.
Some visitors don't read it
User-triggered fetchers may not follow robots.txt, and a badly behaved crawler won't. Blocking that holds happens at the firewall.
How a bot reads the file, and what to write.
Three things to understand before you edit anything.
The most specific group wins, alone.
This is the rule that surprises people. A bot doesn't merge its own group with the general one. It picks one and ignores the other.
- A bot looks for its name firstIf a group names it, that is the only group it follows.
- The general group is a fallbackIt applies only to bots with no group of their own.
- So repeat what's sharedIf /admin/ should be closed to everyone, say so in each named group.
User-agent: * Disallow: /admin/ User-agent: GPTBot Disallow: /blog/ # GPTBot follows only its own group, # so /admin/ is open to it.
Three kinds of bot, three kinds of rule.
Before writing a rule, know which kind of bot it's for. Blocking the wrong kind has the opposite effect from the one you wanted.
- Search botsBuild the index an AI engine answers from. Block them and you leave that engine's results.
- Training botsCollect content for future models. Blocking is a choice about your content.
- User-triggered fetchersOpen a page because a person asked. robots.txt may not apply to them.
A pattern for the common choice.
Many sites want to appear in AI search and stay out of model training. This is one way to say so. Whether it's right for you is a business decision.
- Several names can share a groupList the user-agents one after another, then the rule.
- Keep search and training apartIn separate groups, so a change to one can't touch the other.
- Test each bot against live URLsBefore the file goes live, and again after each release.
User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: / User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended Disallow: /
Writing the file without surprises.
Most of the work is deciding. The typing takes a minute.
Decide your policy
For each kind of bot: search, training, user-triggered. Write it down in a sentence.
List the bots you'll name
Use each company's own documentation for the current names.
The bots table →Write the groups
Repeat any shared rules inside each named group.
Test before publishing
Check live URLs against each bot, and confirm which group applied.
Free robots.txt diff tracker →Check the layer in front
A firewall or CDN can block a bot whatever the file says.
Re-check after releases
A deploy can replace the file, or add a rule nobody reviewed.
Crawlability →Write the rules without the mistakes.
Pick a stance for each AI crawler and the generator writes the groups for you. After a release, compare the live file with the version you saved.
The names to use, and whether they listen.
| Token | Run by | Kind | Follows robots.txt? |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Search | Yes. |
| GPTBot | OpenAI | Training | Yes. |
| ChatGPT-User | OpenAI | User-triggered | OpenAI says the rules may not apply. |
| Claude-SearchBot | Anthropic | Search | Yes. |
| ClaudeBot | Anthropic | Training | Yes. |
| Claude-User | Anthropic | User-triggered | Anthropic says it does. |
| PerplexityBot | Perplexity | Search | Yes. |
| Perplexity-User | Perplexity | User-triggered | Generally not, because a person started the request. |
| Google-Extended | Control token | It is a setting, with no crawler behind it. | |
| Googlebot | Search, and AI Overviews | Yes. | |
| Bingbot | Microsoft | Search, and Copilot | Yes. |
Checked against OpenAI's and Anthropic's documentation, and reporting of Perplexity's and Google's, on 4 October 2026. Other AI bots exist, such as Applebot-Extended and CCBot. Check each company's own page before relying on a name.
The rule behind every block.
Technical Suite tests each bot against your pages on every crawl and shows the exact line responsible when a block looks unintended. It never changes robots.txt for you, because whether to block a bot is your decision.
Frequently asked questions.
The main crawlers from OpenAI, Anthropic, Perplexity, Google and Microsoft say they do. User-triggered fetchers are a different matter: OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person started the request.
Allow the search bots, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, and disallow the training bots, such as GPTBot and ClaudeBot. Put them in separate groups.
Because a bot with its own group follows only that group. It doesn't also read the general rules. Repeat anything shared inside the named group.
No. It governs future crawling. It doesn't recall anything gathered before the rule was in place.
That's a business decision. Blocking the search bots takes you out of AI search results. Blocking the training bots protects your content and means future models learn less about you from your own pages.
No. It's a token you can use in robots.txt to control how Google uses your content for Gemini. It doesn't affect Google Search.
Yes. The SEORCE AI crawler robots.txt generator writes allow and block rules for each AI bot, from a preset or bot by bot. It's free and needs no account.
Find the rule you didn't mean to write.
Add your site and run a crawl. Each AI bot is tested separately, and you'll see the line behind any block.
Not ready for an account? Use the free browser extension or any of the free tools.