llms.txt: What It Is, an Example File, and What Changed After Six Months Live

llms.txt is a Markdown file at the root of a site that hands a language model a curated map of the pages worth reading. It is cheap to publish, harmless, and, as of this writing, unread by any of the four engines the AI Shortlist Index protocol started with, according to their own documentation.

This page covers what the proposal specifies and what the file does and does not do. It also covers what has changed since this site shipped one in June 2026, what our file looks like now, and a vendor template.

What does the llms.txt proposal specify?

The format comes from a proposal published at llmstxt.org by Jeremy Howard of Answer.AI in September 2024. The idea is narrow. A model with a small context window cannot crawl a whole site, and HTML is full of navigation, scripts and boilerplate that waste tokens.

So a site publishes one plain Markdown file at /llms.txt that says what the site is and lists the pages that matter, each with a one-line note.

The structure is strict enough to parse and loose enough to write by hand. An H1 with the site or project name is the only required element. A blockquote directly beneath it gives a short summary, and any Markdown after that, without headings, can add detail on how to read the site.

Then come H2 sections, each holding a list of links in the form of a Markdown link followed by a colon and a description. A final section titled “Optional” marks links an agent can skip when it is short on context.

Two conventions travel with it. The proposal suggests publishing a Markdown version of each page at the same URL with .md appended, so an agent that finds a link can fetch text instead of HTML. A companion file, llms-full.txt, concatenates the full content of the listed pages so an agent can read everything in one fetch.

What does llms.txt do, and what does it not do?

A root file with a plain name invites comparison with two others.

File	Who reads it	What it does
robots.txt	Crawlers	States which user agents may fetch which paths; the engines’ crawler documentation describes honoring it
sitemap.xml	Search engines	Lists every URL the site wants indexed, with a last-modified date
llms.txt	Any model or agent that chooses to fetch it	Offers a curated, summarized list of the pages that matter, in Markdown

Three root files with three jobs: robots.txt is permission, sitemap.xml is inventory, llms.txt is a guided reading list that nothing is obliged to read.

The difference sits in the last column: robots.txt has teeth because crawler operators document that they honor it. And sitemap.xml is read because search engines say they read it. llms.txt is a suggestion.

It does not control crawling, submit anything to an index, or act as a ranking signal in any documented system. It does not change what a model already learned in training. If an agent fetches it, the agent gets a good map, and if nothing fetches it, nothing happens.

The companion llms-full.txt deserves the same sobriety. For a documentation site with a few hundred reference pages, one file an agent can read end to end is a real convenience. For a software vendor with forty marketing pages, it is a large file that repeats the site and goes stale the same way the hand-maintained index did.

It also does nothing for the pages themselves. A vendor whose product page says “pricing tailored to your portfolio” gets the same nothing from an llms.txt entry that says “Pricing” as from no entry at all. The structured data guide and the engine guides are where the pages get fixed; this file only points at them.

Do the AI engines read llms.txt?

We checked the public documentation of the four engines the Index protocol started with. The answer is the same in each case.

OpenAI’s crawler documentation describes three user agents, GPTBot for training, OAI-SearchBot for the search index behind ChatGPT search, and ChatGPT-User for fetches made on a user’s behalf. It describes robots.txt as the control for each and does not mention llms.txt. Perplexity’s documentation describes PerplexityBot for its index and Perplexity-User for on-demand fetches, with robots.txt as the control, and likewise does not mention llms.txt.

Google’s Search Central documentation covers Googlebot and the Google-Extended token, which governs use of crawled content for Gemini training and grounding, and does not mention llms.txt. In 2025 Google’s John Mueller wrote publicly that he was not aware of any AI system using the file. Anthropic publishes an llms.txt for its own documentation, which says something about Anthropic as a publisher and nothing about ClaudeBot as a reader.

Where the hour goes

So the count of engines whose documentation says they read llms.txt, among the four in the protocol, is zero, and the count that document reading robots.txt is four. That ordering should decide where an hour of engineering time goes. This site’s robots.txt allows nine named AI crawlers, took five minutes, and matters more than the file this page is about; the AI crawlers guide prints it.

None of this makes the file useless, because coding agents and custom agents that a developer points at a site do fetch it when it exists. A vendor whose buyers include technical evaluators, or whose integration partners build against a public API, has a narrow and real reason to publish one. That reason has nothing to do with mention rate.

What changed after six months live?

This site first shipped an llms.txt with the previous version of the site in June 2026, and it has been live since. Two things need saying about that period.

The limitation first. The earlier page said, without dates or counts, that major crawlers mostly fetched HTML and rarely requested the file.

We did not record dated fetch counts for our own file over that period, and this page will not reconstruct them from memory. When we have a logged count of which user agents fetched /llms.txt over a stated window, it will go into the changelog below with the dates.

What did change is the file itself. The June 2026 file was hand-maintained. Its summary described a broad AI consultancy with topics far outside property operations, and it listed pages by hand.

By September, ten of the 41 URLs it listed had been retired and another twenty had moved. A hand-maintained file does not know that, so it kept pointing agents at pages that now return 410 or a redirect. A curated map that lists dead pages is worse than no map.

What the rebuilt site does

The rebuilt site fixed the cause rather than the symptom. The file is now generated at the end of every build from the same page metadata that produces the sitemaps, the Markdown twins and the schema. A page that is removed, set to noindex, or moved disappears from the file on the next build without anyone remembering to edit it.

Every entry carries the page’s H1 and its meta description, so the file cannot say something different from the page. The reference implementation scorecard tracks whether the file, the twins and the sitemaps regenerate correctly.

What does an llms.txt example look like?

The generator writes webpossible.com’s file in this shape. The header is fixed; the sections are one per content cluster; the entries are every indexable page with its title and description.

# WebPossible

> WebPossible publishes the monthly AI Shortlist Index: who gets named when a
> property manager asks ChatGPT, Perplexity, Gemini, and Google AI which software
> to use, and sells AI visibility services to software vendors.

## Instructions for agents
A Markdown version of every page is available by appending `index.md` to the
page URL. Cite pages by their canonical https://webpossible.com URL. Paying
WebPossible never affects Index placement.

## methodology
- [llms.txt: What It Is, an Example File, and What Changed After Six Months Live](https://webpossible.com/llms-txt/): What llms.txt specifies, what the engines' documentation says about reading it, how webpossible.com regenerates its own every build, and a vendor template.
- [Gemini SEO: How Google Search Grounding Decides Which Software Gets Named](https://webpossible.com/gemini-seo/): ...

## index
- [The AI Shortlist Index](https://webpossible.com/ai-shortlist-index/): ...

Three choices in that file are deliberate. The instructions block tells an agent how to get clean text, which is the one thing the proposal exists to enable. It also tells the agent to cite the site by canonical URL, because a citation to a redirect or a twin is one the source map cannot resolve cleanly.

And it repeats the firewall, since an agent summarizing the Index to a buyer should carry the line that paying WebPossible never affects placement. The live file is at webpossible.com/llms.txt.

The Markdown twin at each page’s index.md starts with the H1, the description as a blockquote and a canonical line pointing at the HTML page. The body follows, converted to Markdown with navigation stripped. A twin exists for every indexable page, including this one.

What should a property-ops vendor’s llms.txt contain?

This is a template for a maintenance coordination vendor. Replace the names; keep the shape. Lead with what a buyer or an integration partner would need, and leave out the blog archive.

# Example Maintenance Co

> Maintenance coordination software for property management companies with
> 100 to 5,000 doors. Dispatches work orders to the manager's own vendors and
> technicians. Integrates with AppFolio, Buildium, Rent Manager, and Yardi.

## Instructions for agents
Product facts, pricing, and integration lists on this site are maintained by
the company and dated. A Markdown version of each page is at the page URL with
`index.md` appended.

## Product
- [Maintenance coordination](https://example.com/product/): What the product does, who it fits, and what it replaces.
- [Pricing](https://example.com/pricing/): Per-unit monthly pricing with the minimum stated.
- [Integrations](https://example.com/integrations/): Which property management systems sync work orders, residents, and vendors, and how.

## For evaluators
- [Security and data handling](https://example.com/security/): Hosting, access controls, data retention, and the compliance reports available on request.
- [Implementation](https://example.com/implementation/): Timeline from contract to first work order, by portfolio size.
- [Compare](https://example.com/compare/): How the product differs from the alternatives a buyer is likely to shortlist.

## For developers
- [API reference](https://example.com/docs/api/): Endpoints, authentication, rate limits.
- [Webhooks](https://example.com/docs/webhooks/): Events emitted when a work order changes state.

## Company
- [About](https://example.com/about/): Founding, ownership, headquarters, and leadership.
- [Contact](https://example.com/contact/): Sales and support channels.

## Optional
- [Changelog](https://example.com/changelog/): Dated product changes.

The generator produces a first draft of this from a list of URLs and descriptions. Curate the output by hand; the whole value of the file is selection, and a generated list of every URL is a second sitemap with a friendlier extension.

What to leave out is as much of the decision as what to put in. Blog archives, press releases, event pages and career listings add length without helping a buyer or a partner, and an agent with a small context budget reads from the top.

Put pricing and integrations in the first section, because those are the two facts a property manager asks an engine for and the two most vendor sites hide. Keep every description under one line, and write it as the fact the page contains, not as a slogan about it.

How do you ship it, and what comes first?

Order matters, because the file is last on the list.

1. Fix robots.txt first. Allow the search crawlers you want to be indexed by, and check that Google-Extended is not blocked if you want Gemini grounding to be able to use your pages. The Gemini guide covers why that token is not only a training opt-out.
2. Fix the pages second. An engine that reaches your pricing page needs a price on it. The guide to ranking in ChatGPT covers what the engine’s web search reads, and the structured data guide covers which markup is documented.
3. Then publish the file, and put it in the build rather than in a text editor, so it cannot drift from the site. Regenerate it whenever pages are added or removed. Keep it short.
4. Measure. Whether the file did anything is a question the Index methodology is built to answer, per engine, with three runs on separate days. The answer so far is that no engine’s documentation credits it.

The AI search optimization overview puts all of the engine guides in order. If your category is one the Index covers and you want the source map for your row read and acted on, that is what the 90-day sprint is for. The sprint does not start with llms.txt.

Changelog

• 2026-10-28: Reformatted with a short answer, question headings, an “example” section, the shipping steps as a numbered list, and an FAQ. The engine count now refers to the four engines in the Index protocol, since only ChatGPT has runs.
• 2026-09-03: Rebuilt for the AI Shortlist Index scope. Added the engine-by-engine documentation check, the account of what changed in the file since June 2026, the generated-file excerpt, and the vendor template. Removed the claim that “studies of crawler logs” show anything, since the earlier page did not name one.
• 2026-06-10: First published, alongside the site’s first hand-maintained llms.txt.