llms.txt
A simple markdown file that hands AI agents a clean map of your site instead of making them crawl your HTML. Here is what it is, how it differs from robots.txt, whether anyone reads it yet, and how to ship one.
Definition
What llms.txt is
llms.txt is a plain markdown file at the root of your site that gives a language model a curated, summarized map of your best content. Instead of crawling and parsing a sprawl of HTML, an agent can read one clean document and understand what you do and where to look.
The format is deliberately simple: a title, a one-line summary of your site, then sections of links with a short description for each. You are doing the model's first pass of work for it, and pointing it at the pages you actually want it to use. This is one piece of the broader agent legibility stack.
The lineup
llms.txt vs robots.txt vs sitemap.xml
Three root files, three different jobs. You want all three.
| File | Audience | Job |
|---|---|---|
robots.txt | Crawlers | Permission: what may be accessed |
sitemap.xml | Search engines | Inventory: every URL to consider |
llms.txt | Language models | A guided tour: the best pages, summarized in markdown |
A larger companion file, llms-full.txt, takes it further by concatenating the full markdown of your key pages so an agent can ingest everything in one fetch. We publish our llms.txt and our llms-full.txt, because a site that sells agent legibility without one would be a bad joke.
Straight talk
Does it actually work yet?
Here is the honest answer most pages will not give you. As of now, the major training and search crawlers mostly ignore llms.txt and fetch your HTML directly. There is no public commitment from the big labs to read it in production, and studies of crawler logs show requests to llms.txt are still rare. What does use clean machine versions today are coding agents like Claude Code and Cursor, which request markdown when it is offered.
So why ship it? Because it costs almost nothing, it helps the agents that do read it, it forces you to think clearly about your most important pages, and models train on the conventions that exist now. It is cheap insurance and a small head start, not a silver bullet. We would rather tell you that than sell you a myth. The higher-leverage work is structured data and entity consistency.
How to create your llms.txt
List your most important pages, grouped into a few sections, each with a one-line description in plain markdown. Lead with what you want agents to use most. Keep it curated rather than exhaustive, because the point is guidance, not a second sitemap. Our free llms.txt template and generator gives you an annotated starting point and a draft you can refine by hand.