USE CASE
AI crawler optimization,
checked bot by bot.
An AI engine can only cite a page its crawler could reach and read. This guide covers which bots matter, what stops them, and how to check your own site. SEORCE tests access for each bot separately and shows you which ones visit.
What is AI crawler optimization?
AI crawler optimization is the work of making sure the bots behind AI answer engines can reach your pages, read what is on them and understand what they are about. In practice that means access rules such as robots.txt, content that is present in the raw HTML, clear structure such as headings and schema, and knowing which AI bots visit. It builds on technical SEO. The main difference is that there are many bots, and most of them behave differently from Googlebot.
Key facts
- AI companies run several bots each, with separate jobs: training models, building a search index, and fetching a page when a user asks
- Blocking a training bot and blocking a search bot have different effects. OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers
- Many AI crawlers do not run JavaScript, so content that only appears after scripts load may never be seen
- One robots.txt rule can allow one bot and block another, often by accident
- llms.txt is a proposed convention. Treat it as optional, never as a replacement for access and readable HTML
- Robots.txt is not the only gate. A firewall or CDN can block a bot first, and a user agent can be faked, so access and visits both need checking
- SEORCE checks access for each bot separately, compares raw HTML with the rendered page, and shows visits from AI crawlers and other bots
Last reviewed 15 October 2026 by the SEORCE team.
AI bots can't always read what Google can.
Often they can't. Most of what we know about crawling comes from Googlebot, which renders JavaScript and reports back through Search Console. AI crawlers do neither reliably, and nothing tells you when they've failed to read a page.
What you know about Googlebot
- –It renders JavaScript before reading the page
- –One main crawler, well documented
- –Search Console reports what it found
- –Your robots.txt was written with it in mind
What's true of AI crawlers
- ✓Many read the raw HTML and nothing more
- ✓Several bots per company, each with its own job
- ✓Little or no reporting on what they saw
- ✓A rule written for one can block another
Four checks decide whether you're read.
Work through them in this order, because there's no point polishing a page a bot was never allowed to open.
Each bot can reach the page
Access is decided bot by bot. A page that is open to Googlebot can be closed to the crawler behind ChatGPT search, and nothing on the page will tell you.
- Rules for each botCheck robots.txt for every crawler you care about. A general "allow" can be overridden by a rule aimed at one bot.
- Training bots and search botsThey are different crawlers. You can keep content out of training and still appear in AI search, if the rules are written with care.
- Blocks nobody meantA rule written for a staging folder, or copied from another block, can cover live pages.
- Blocks before robots.txtA firewall, CDN or bot-protection setting can turn AI bots away before they read your rules. The bot gets an error page, and robots.txt never comes into it. Check the response each bot receives, as well as the rules.
The content is in the HTML
A bot that doesn't run JavaScript sees only what the server sends. If your pricing or your answers arrive later by script, that bot reads a page without them.
- Raw HTML against the rendered pageCompare what the server sends with what a browser shows. The difference is what a non-rendering bot misses.
- The important parts firstPut the answer, the facts and the main links in the HTML itself. Tabs and accordions that load on click are a common trap.
- A quick, clean responseSlow pages, long redirect chains and error codes give a bot reasons to leave early.
A machine can tell what the page is
Reading a page and understanding it are two steps. Clear structure helps a bot work out what the page covers and which parts answer a question.
- Headings that outline the pageOne H1, then headings in order. A bot uses them the way a reader uses a contents list.
- Valid structured dataSchema states plainly what the page is about. Broken or missing markup removes that help.
- An accurate sitemapIt should list the pages that exist and leave out the ones that don't.
- Internal links to every pageA page nothing links to is hard for any crawler to find.
The bots that came
The first three checks tell you what should happen. Visits tell you what did. They are the only direct evidence that a crawler is reading your site.
- Visits by botSEORCE shows visits from AI crawlers and from other bots, in Web Analytics and in AI Beacon, so you can see who comes and how often.
- Verified bots and impostorsA user agent is easy to fake. OpenAI and Perplexity publish the IP ranges their bots use, so a visit can be checked against them.
- The pages they readCompare the pages bots fetch with the pages you want cited. The gap is your to-do list.
- What followsSet bot visits beside AI referral traffic and citations. A bot visit is the start of the chain, never proof of a citation.
Different bots, different jobs.
Most confusion about blocking AI comes from treating every bot as the same thing. Each company runs several, and blocking one does not block the others.
| Bot | Run by | Its job | If you block it |
|---|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train models. | Your content stays out of training. ChatGPT search is not affected. |
| OAI-SearchBot | OpenAI | Builds the index behind ChatGPT search. | OpenAI says your site won't be shown in ChatGPT search answers. |
| ClaudeBot | Anthropic | Collects content that may be used to train models. | Your content stays out of training. |
| Claude-SearchBot | Anthropic | Indexes pages for search inside Claude. | Your pages are less likely to appear when Claude searches. |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity's search results. Perplexity says it is not used to train models. | Your pages are less likely to be cited in Perplexity. |
| Googlebot | Crawls for Google Search, which AI Overviews draw on. | You leave Google Search as well, which is rarely what anyone wants. | |
| Google-Extended | A robots.txt token, with no crawler of its own. It controls use of your content for Gemini. | Google Search is not affected. | |
| Bingbot | Microsoft | Crawls for Bing, which Copilot draws on. | You leave Bing, and with it Copilot's answers. |
| User-triggered fetchers | Several | ChatGPT-User, Claude-User and Perplexity-User open a page when a person's request needs it. | It varies. Anthropic says Claude-User follows robots.txt. The other two act for a person, and robots.txt may not apply to them. |
How to optimise for AI crawlers, in six steps.
These are the same six steps as any technical SEO work. The difference is that you run them once for each bot.
Test access as each bot
Check every important page against the rules for each crawler. "Crawlable" on its own tells you nothing about which bot.
Free AI crawler log analyser →Crawlability →Find the rule or the script
For each gap, find the cause: the exact robots.txt line, the firewall setting, or the script that loads the content late.
Free robots.txt conflict checker →Set your policy, then rank the gaps
Decide which bots you want. Then fix the pages you most want cited first: product, pricing and the pages that answer buyer questions.
Change rules by hand, structure by machine
Edit robots.txt yourself, because blocking is a decision. Let software repair schema, headings and internal links.
AutoFix →Test again as each bot
Re-run the access check and the raw HTML comparison. A fix for one bot can be undone by a rule for another.
Site audit →Watch who visits, and what follows
Track bot visits, then the AI referral traffic and citations that should come after them.
AI visibility tracking guide →Start with what your logs already show.
Your server logs record every AI bot that visits and every page it fetches. The log analyser reads them in your browser, and the other tools cover the rules and files behind the result.
One copied rule, and what it cost.
Ledgerly is our demo account, a B2B accounting tool for agencies. The team blocked GPTBot to keep content out of training. Someone copied the block one line too far.
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: * Allow: /
Access was tested for each bot
Googlebot and PerplexityBot could reach every page. OAI-SearchBot could reach none.
Two causes turned up
The exact robots.txt rule was shown. The raw HTML check also found the pricing table arriving by script.
The block came first
A site-wide block outranks everything else. Pricing came second, because buyers ask AI about price.
One line removed by hand
The team deleted the copied rule and kept the GPTBot block. AutoFix repaired malformed schema on 38 pages.
The next crawl confirmed it
OAI-SearchBot showed as allowed on every page. The pricing table went to a developer to move into the HTML.
Then the team watched visits
Bot visits show whether OAI-SearchBot starts arriving. AI Beacon shows whether ChatGPT starts naming Ledgerly.
We've left the outcome out. How soon a bot comes back, and what the engine does then, is outside anyone's control.
What to measure, bot by bot.
Access and readability can be measured before any bot arrives. Visits and traffic tell you what happened afterwards.
| Metric | Check it counts | What it tells you | Read it with care |
|---|---|---|---|
| Pages reachable, per bot | Access | The share of important pages each bot is allowed to fetch. | A deliberate block is a choice. Count the unintended ones. |
| Raw-to-rendered gap | Readability | How much content, and how many links, only appear after JavaScript runs. | Some gaps are harmless, such as a chat widget. Look at what is missing. |
| Schema errors and orphan pages | Structure | How many pages give a machine no clear description, or can't be found by links. | Valid schema helps a bot parse a page. It doesn't earn a citation by itself. |
| AI crawler visits, by bot | Visits | Which bots come, how often, and which pages they read. | Training bots crawl widely. A visit from one is no sign of a citation. |
| AI referral visits | Visits | People arriving on your site from an AI answer. | A mention with no link sends nobody. Check visibility as well as traffic. |
Checked bot by bot. Fixed where it's safe.
A robots.txt checker tells you one thing on one day. Rules change, releases happen and new bots appear, so SEORCE checks again on every crawl, repairs what has one right answer, and shows you who visits afterwards.
Every module talks to the others.
Being readable is the first link in a chain. A bot that can read your page still needs a reason to use it, and you still need to know whether it did.
In SEORCE the chain is already joined up. Technical Suite checks that bots can read the page. Content Studio makes it worth quoting. AI Beacon shows whether engines cite it, and Web Analytics shows the visits that follow.
Ask SEORCE sits on top. It is your AI agent, in the app and on WhatsApp. Ask it a question in plain language and it answers from your project's data.
To start, sign up, link your WhatsApp number and text "Hi" to +1 (363) 200-3169.
Meet Ask SEORCE →Product pages and free tools.
This page covers the subject as a whole. The pages below show how each part works inside SEORCE. The free tools need no account.
AI crawler log analyser
Upload a server log and see which AI crawlers visit, and what they fetch.
Open the log analyser →AI crawler robots.txt generator
Set allow or block rules for each AI crawler.
Open the generator →llms.txt generator
Build an llms.txt file, if you choose to publish one.
Open the llms.txt generator →Crawlability
Sitemap, robots.txt and AI crawler access, checked for each bot.
See crawlability →Site audit
Raw HTML compared with the rendered page, across the whole site.
See site audit →Structured data
Schema validated and repaired where the fix is clear.
See structured data →Internal links
Orphan pages found and linked back into the site.
See internal links →Errors and redirects
Chains, loops and error codes that send a bot away.
See errors and redirects →AutoFix
What is applied automatically, and what waits for a person.
See AutoFix →Web Analytics
The visits AI sources send you, free on every plan.
See Web Analytics →AI visibility tracking
The guide to measuring whether AI engines name and cite you.
Read the tracking guide →Automated technical SEO
The guide to what can be fixed automatically, and what can't.
Read the automation guide →Frequently asked questions.
AI crawler optimization is the work of making sure the bots behind AI answer engines can reach your pages, read what is on them and understand what they are about. It covers access rules, content in the raw HTML, clear structure, and knowing which AI bots visit.
Partly. The foundations are the same: crawlable pages, clean redirects, valid schema. Two things differ. There are many bots, each with its own rules, and many of them do not run JavaScript the way Googlebot does.
That depends on what you want. To appear in AI answers, allow the search bots: OAI-SearchBot, PerplexityBot, Claude-SearchBot, plus Googlebot and Bingbot. Whether to allow training bots such as GPTBot and ClaudeBot is a separate business decision.
They are two separate OpenAI crawlers. GPTBot collects content that may be used to train models. OAI-SearchBot builds the index behind ChatGPT search. OpenAI treats the two settings independently, so you can block one and allow the other.
No. GPTBot is about training. Appearing in ChatGPT search depends on OAI-SearchBot, and OpenAI says sites that opt out of that bot won't be shown in ChatGPT search answers.
Many do not. They read the HTML the server sends and stop there. Content that only appears after scripts run, such as a pricing table loaded on the client, may never be seen. Comparing raw HTML with the rendered page shows what is at risk.
Yes. A rule written for one purpose can match more than was intended, and a block copied from one bot to another is easy to miss. Checking access for each bot separately is the only reliable way to catch it.
No. AI Overviews draw on Google Search, which Googlebot crawls. Google-Extended is a robots.txt token with no crawler of its own. It controls how your content is used for Gemini and does not affect Google Search.
No. llms.txt is a proposed convention, and publishing one is optional. It never replaces access and readable HTML. If you want one, the free llms.txt generator builds it.
Yes, as a parsing aid. Valid structured data states plainly what a page is about, which helps any machine that reads it. It does not earn a citation by itself.
Start with the free Browser extension, which shows whether AI crawlers are allowed on the page you're viewing. For every page on every crawl, crawlability in SEORCE checks access for each bot and shows the exact rule behind a block.
In Web Analytics and in AI Beacon. SEORCE shows visits from AI crawlers and from other bots, including which pages they read. Without a tool, the same information sits in your server logs.
No. Whether to block a bot is a decision for your team. SEORCE shows which rule causes an unintended block and what it affects, and leaves the edit to you.
The structural problems that have one correct answer. AutoFix repairs issues such as schema errors, meta tags, headings and image alt text on the live site. Google and AI crawlers read the fixed version of the page, and each fix type is tested against them before release. Access rules and server settings stay with your team.
Not necessarily. Being readable is required, and it is only the first step. Whether an engine names you depends on the content and on the other sources it trusts. The AI visibility tracking guide covers how to measure that.
Ask SEORCE is SEORCE's AI agent, currently in beta. It works in the app and on WhatsApp, and answers plain-language questions from your project's data, such as which AI crawlers are blocked. It is included for every user, with no usage limit. To use it on WhatsApp, link your number and text Hi to +1 (363) 200-3169.
Yes. The SEORCE extension for Chrome, Edge and Firefox checks the page you're on: whether AI crawlers are allowed, what the raw HTML contains and how the page is structured. It's free and needs no account.
Find out which bots can read your site.
Add your site and run the first crawl. You'll see access for each bot, the rule behind every block, and what the raw HTML is missing.
Not ready for an account? Use the free browser extension or any of the free tools.