GUIDE · PART OF AI CRAWLER OPTIMIZATION
GPTBot and OAI-SearchBot:
which to allow, which to block.
OpenAI runs more than one bot, and they do different jobs. Block the wrong one and you drop out of ChatGPT search while your content still goes into training, which is the opposite of what most people intend. This guide sets out what each bot does, with the robots.txt rules for the common choices.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot and OAI-SearchBot are two separate web crawlers run by OpenAI. GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot builds the index that ChatGPT's search features draw on. Each follows its own robots.txt rule, so a site can allow one and block the other. A third agent, ChatGPT-User, fetches a page when a person's request calls for it.
Key facts
- They are two crawlers with two jobs, and each has its own robots.txt rule
- OpenAI says sites opted out of OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links
- Blocking GPTBot keeps your content out of training. It has no effect on ChatGPT search
- If you allow both, OpenAI says it may use a single crawl for both purposes
- OpenAI says a robots.txt change can take about 24 hours to show in search
- OpenAI publishes the IP ranges each bot uses, so a visit claiming to be one of them can be checked
Last reviewed 15 October 2026 by the SEORCE team.
"The ChatGPT bot" is at least three bots.
People talk about OpenAI's crawler as if there were one. There are several, and most mistakes come from writing a rule for one while thinking of another.
Training, search and user requests
A rule that names GPTBot does nothing to OAI-SearchBot, and the other way round.
"AI bots to block"
Lists passed around online often include the search bot. Paste one in and you've opted out of ChatGPT search without meaning to.
One crawl can serve both
Because OpenAI may reuse a crawl, you can't always tell from your logs which purpose a visit served.
Two bots, and the choice most sites make.
Whether to allow training is your decision. The mechanics below are the same whichever way you go.
The training crawler.
GPTBot gathers content that may be used to train OpenAI's models. Allowing it means your pages can inform what future models know. Blocking it keeps them out.
- What blocking doesYour content isn't collected for training from then on.
- What it doesn't doIt has no effect on whether you appear in ChatGPT search.
- The token to useGPTBot, in its own group in robots.txt.
User-agent: GPTBot
Disallow: /The search crawler.
This is the one that decides whether ChatGPT search can show your pages. If you want to be found there, it needs to get in.
- Allow it in robots.txtOpenAI recommends allowing it if you want to appear in search results.
- Allow its IP ranges tooA firewall that blocks them undoes the robots.txt rule.
- Give changes a dayOpenAI says updates can take about 24 hours to take effect in search.
- Opted-out sitesWon't be shown in ChatGPT search answers, though they may still appear as navigational links.
User-agent: OAI-SearchBot
Allow: /In search, out of training.
This is the setup many sites settle on. It keeps you visible in ChatGPT search and keeps your content out of model training. It is one option among several, and the choice is yours.
- Two groups, one for each botEach bot reads only the group addressed to it.
- Keep them apartDon't list the two bots under one Disallow.
- Check the resultTest each bot against a live URL before you publish the file.
User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: /
Setting your rules for OpenAI's bots.
Five steps. The second is the one that turns up surprises.
Decide your policy
Do you want to be in ChatGPT search? Do you want your content used for training? They are separate questions.
Check what you have now
Test your domain against each bot. Many sites find a block they didn't know about.
Free AI crawler log analyser →Write a group for each bot
One for OAI-SearchBot, one for GPTBot. Test both before publishing.
robots.txt for AI crawlers →Look at the layer in front
Make sure a firewall or CDN isn't turning the search bot away before it reads the file.
See how OpenAI's bots are treated on your site.
Your server logs show whether GPTBot and OAI-SearchBot visit, and how often. The generator then writes the robots.txt rules for each bot, so you can allow one and block the other.
OpenAI's agents, and what blocking each does.
| Agent | Its job | If you block it |
|---|---|---|
| OAI-SearchBot | Builds the index behind ChatGPT search. | OpenAI says your site won't be shown in ChatGPT search answers, though it may still appear as a navigational link. |
| GPTBot | Collects content that may be used to train models. | Your content stays out of training. Search is not affected. |
| ChatGPT-User | Opens a page when a person's request needs it. | OpenAI says robots.txt rules may not apply, because a person started the request. |
Checked against OpenAI's crawler documentation, 4 October 2026. OpenAI changes these details from time to time, so read its page before you edit your file.
Each bot checked, on every crawl.
Technical Suite tests access for OAI-SearchBot and GPTBot separately each time it crawls, and shows the exact rule behind an unintended block. It never edits robots.txt for you. SEORCE also shows visits from AI crawlers, so you can see whether the bot you allowed has started arriving.
Frequently asked questions.
GPTBot is OpenAI's training crawler. It collects content that may be used to train OpenAI's models. You control it with a GPTBot group in robots.txt.
OAI-SearchBot is the OpenAI crawler that builds the index behind ChatGPT's search features. OpenAI recommends allowing it if you want your site to appear in ChatGPT search results.
No. GPTBot is only about training. Whether you appear in ChatGPT search depends on OAI-SearchBot, which has its own rule.
Yes. Allow OAI-SearchBot and disallow GPTBot, each in its own group. OpenAI treats the two settings separately.
OpenAI says it can take about 24 hours for a robots.txt update to be reflected in search.
Check its IP address against the ranges OpenAI publishes for each bot. A user agent alone can be faked.
Yes. The free browser extension shows whether AI crawlers are allowed on the page you're viewing, and the AI crawler log analyser shows whether each bot is reaching your site. Neither needs an account.
Check OpenAI's bots on your site.
Add your site and run a crawl. You'll see whether each bot is allowed, and the rule responsible where one isn't.
Not ready for an account? Use the free browser extension or any of the free tools.