GUIDE · PART OF AI CRAWLER OPTIMIZATION
llms.txt:
what it is, and whether you need one.
llms.txt is a small file that lists a site's most useful pages for AI systems to read. It has been talked about far more than it has been used. Whether you need one has a plain answer: probably not yet, and publishing it does no harm. This guide covers what it is, what the evidence says, and how to write one if you decide to.
What is llms.txt?
llms.txt is a proposed convention for a plain Markdown file, placed at the root of a site, that gives AI systems a short, curated list of the site's most important pages with a line about each. It was proposed in 2024 and is not an official standard. It doesn't control what a crawler may fetch, which is the job of robots.txt. Its clearest use so far is on developer documentation, where AI coding assistants read it to find the right pages.
Key facts
- llms.txt is a proposal, published in 2024. No standards body has adopted it
- It is a reading list for AI systems. It grants and blocks nothing
- Google's guidance says you don't need to create new machine-readable files to appear in Search or its AI features
- Studies published in 2026 found that the major AI crawlers rarely request the file
- Its proven use is developer documentation, read by AI coding assistants
- It costs little to publish, and there is no good evidence that it raises AI citations
Last reviewed 15 October 2026 by the SEORCE team.
A small file with a lot of claims attached.
Much of what is written about llms.txt comes from people selling a way to make one. The file is harmless and can be useful. It isn't the key to AI visibility.
"Essential for AI search"
It isn't. A site with no llms.txt and readable pages will do better than one with the file and a blocked search bot.
It doesn't control access
Listing a page in llms.txt doesn't allow a crawler in, and leaving one out doesn't keep a crawler away.
Published, and seldom fetched
Adoption has grown quickly. Signs that the big AI engines read the file for their answers have not.
What goes in it, and what it's good for.
Three things to know before deciding whether it's worth your afternoon.
A short Markdown file at the root.
The proposal is deliberately simple. If you can write a README, you can write an llms.txt.
- A title and a one-line summaryYour name as a heading, then what you do.
- Sections of linksEach link with a short description of the page.
- An optional sectionFor pages that can be skipped when space is short.
- A companion fileSome sites also publish llms-full.txt, with the full text of the pages.
# Ledgerly > Invoicing and accounting software for agencies. ## Product - [Multi-currency invoicing](/features/multi-currency): Invoice in 30 currencies. - [Pricing](/pricing): Plans and what each includes. ## Guides - [Retainer billing](/guides/retainer-billing): How to bill a monthly retainer. ## Optional - [Changelog](/changelog): Product updates.
Know what you're getting.
It helps to be clear about what the file can and can't do, because the claims made for it go well beyond the evidence.
- It points to your best pagesIn a clean form that is cheap for a model to read.
- It doesn't control crawlersUse robots.txt for that.
- It isn't a ranking signalGoogle has said it doesn't use these files.
- It may matter more laterConventions take years to settle. This one is still early.
Worth it for some sites, optional for most.
If you publish documentation that developers use through AI coding tools, write one. For everyone else it sits a long way down the list.
- Developer docs and APIsThe one place it clearly earns its keep.
- Everyone elseOptional. Do access and readable HTML first.
- Keep it currentA stale file that points at dead pages is worse than none.
Writing one, if you decide to.
It takes less time than reading about it.
Decide whether it fits
Strong case if you have developer docs. Otherwise treat it as a nice-to-have.
Pick your best pages
Ten to twenty that explain what you do. This is a shortlist, and a sitemap already lists everything.
Write a line for each
Say what the page covers, the way a directory entry would.
Free llms.txt generator →Publish and validate
Put it at the root of the site and check the format.
Review it now and then
Update it when pages move or products change.
Write one in a few minutes, if you want one.
The generator builds an llms.txt from the pages you choose. Sort out your robots.txt rules for AI crawlers first, because access matters far more.
Three files, three different jobs.
| File | What it's for | Who reads it | Do you need it? |
|---|---|---|---|
| robots.txt | Says which crawlers may fetch which pages. | Search engines and AI crawlers. | Yes. |
| sitemap.xml | Lists your pages so they can be found. | Search engines. | Yes. |
| llms.txt | A curated list of your best pages, in Markdown. | Some AI assistants and coding tools. | Optional. |
Based on Google's published guidance and on studies of crawler behaviour published in 2026. This is changing, so check again before treating it as settled.
Access first, then the file.
Technical Suite checks what decides whether AI bots can use your site: access for each bot, and whether your content is in the raw HTML. The free llms.txt generator covers the file itself, for when you want one.
Frequently asked questions.
llms.txt is a proposed convention for a Markdown file at the root of a site that lists its most important pages for AI systems, with a short description of each.
No. Google's guidance says you don't need to create new machine-readable files to appear in Search, including its AI features, because Search doesn't use them.
OpenAI hasn't said that it does. Its crawler documentation directs site owners to robots.txt. Studies of crawler traffic have found the major AI bots rarely request the file.
There's no good evidence that it does. Being reachable by AI search bots, and having pages that answer questions plainly, matter far more.
No. robots.txt tells crawlers what they may fetch. llms.txt is a reading list. It doesn't allow or block anything.
At the root of your site, so that it loads at yoursite.com/llms.txt.
Yes. The SEORCE llms.txt generator creates a file from the pages you pick, ready to upload. It's free and needs no account.
Check the things that matter more.
Add your site and run a crawl. You'll see whether AI bots can reach your pages, and what they find when they do.
Not ready for an account? Use the free browser extension or any of the free tools.