Markdown Twins for Every Page, With No Proof Yet That Anyone Reads Them

Every indexable page on this site has a Markdown twin. Append index.md to the page URL and you get the same content as plain text, with the navigation, footer and scripts stripped and a line pointing back at the HTML page as the canonical. The twins cost nothing to keep once they were in the build, and they may do nothing at all, which is the part this post is careful about. It covers what the twins are and why we built them, how they are generated, what to watch if you build your own, and what evidence would tell us whether an engine ever reads one.

What a twin is

The page you are reading lives at /blog/markdown-twins/. Its twin lives at /blog/markdown-twins/index.md. The twin starts with the page’s H1, then the meta description as a blockquote, then a canonical line naming the HTML URL, then the body converted to Markdown. Tables stay tables, headings stay headings, links keep their targets. What is gone is everything the layout adds around the content: the header, the breadcrumbs, the schema block, the footer with the firewall, and the scripts.

The site’s llms.txt file tells an agent about them in its instructions block. It says a Markdown version of every page is available by appending index.md to the page URL, and it says to cite pages by their canonical https://webpossible.com URL. Those two sentences are the whole contract. Read the twin if you want clean text; cite the HTML page regardless.

Why we built them

The idea comes from the llms.txt proposal at llmstxt.org, which suggests publishing a Markdown version of each page at its URL with .md appended, so an agent that finds a link in the file can fetch text instead of HTML. This site’s URLs end in a slash, so the convention here is index.md rather than a bare .md suffix. The llms.txt guide covers the proposal itself and what each engine’s documentation says about the file, which is nothing.

The twins solve a different problem from the one covered in how engines read a vendor page. That post is about what the page says. This one is about the wrapper around it.

The case for the twins is a token argument. A page on this site carries a header, a breadcrumb trail, a JSON-LD block, a footer and the styling to hold it together, and a model reading that HTML spends context on all of it before it reaches the lede. The twin is the lede and what follows. For a coding agent or a custom agent a developer points at the site, a document that is only the document is the format they handle best. That is a real use with a narrow audience.

The other case is that the twins were nearly free. They are produced by the same build step that produces the sitemaps, the schema and the llms.txt file, from the same page metadata. Adding them was a few hours once. Maintaining them is nothing, because nothing maintains them by hand.

How they are generated

The twins are written at the end of every build. The generator walks the same list of indexable pages that feeds the sitemaps, and for each one writes the H1, the description, the canonical line and the converted body to index.md alongside the page’s HTML. A page that is removed, set to noindex, or moved disappears from the twin set on the next build, for the same reason it disappears from the sitemap and from llms.txt. There is no separate list to forget.

That is the property that matters. The previous version of this site kept a hand-maintained llms.txt, and by the time the site was rebuilt ten of the 41 URLs in it were gone and twenty had moved. A hand-maintained twin set would rot the same way, and a twin that says something the page no longer says is worse than no twin, because the one agent that does fetch it gets the stale version. Generate or skip.

What to watch

Canonical is the first thing. A Markdown file has no head element, so there is nowhere to put a rel=canonical link tag. The twin carries a canonical line in its text instead, and llms.txt tells agents to cite the HTML URL. Google Search Central documents a second option for non-HTML documents, an HTTP Link header carrying rel=canonical, which a server can send with the Markdown response. If your hosting lets you set headers per path pattern, send it.

Duplicate content is the second, and it worries people more than the documentation supports. Google’s documentation on duplicate content says duplicates are not penalized and that when it finds several versions of the same content it picks one as canonical; the risk it describes is the wrong one being chosen, which is what the canonical signals above exist to prevent. A text file that names its HTML page as canonical, from a site whose llms.txt says the same, is about as clear as the signal gets.

Sync is the third, and the build solves it. Anything a person has to remember to update will be forgotten. If the twin is not generated from the same source as the page, do not publish it.

Crawler access is the fourth. A twin behind a robots.txt rule that blocks the AI crawlers is a twin nobody can fetch. This site’s robots.txt allows the OpenAI, Perplexity, Anthropic and Google AI user agents, and the twins live under the same paths as the pages, so they inherit the same access. The crawler and robots.txt guide lists the user agents and what each one’s documentation says it is for.

What we do not know, stated as a hypothesis

We have no evidence that any of the four engines the Index measures fetches a Markdown twin. The crawler documentation from OpenAI, Perplexity and Google describes fetching pages and honoring robots.txt, and none of it mentions the .md convention or llms.txt at all. The one Index run so far, a single ChatGPT answer to prompt mvn-best-01 on 2026-09-03, cited 124 sources and none of them was on webpossible.com, in HTML or in Markdown.

Our hypothesis is that the twins are read by agents developers point at the site and by nothing else, and that they contribute nothing to whether an engine names a vendor. We publish them anyway, for the narrow audience above and because the cost is zero, and we would rather say that plainly than imply a mechanism the documentation does not describe.

Three things would change the answer. The first is a server log over a stated window showing OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User or Google-Extended requesting an index.md path. The second is a citation in an Index run file whose URL ends in index.md; the source map flags any webpossible.com URL as own network and discloses it inline, so it would not be missed. The third is a citation to an HTML page on this site that appears in an answer whose text matches the twin’s conversion rather than the page’s rendering, which would be weak evidence and would be labeled as such. When any of the three happens, it goes into the changelog on the llms.txt guide with the dates.

Should a vendor do this

Last, and only if it is free. The order for a property-ops software vendor is robots.txt first, so the search crawlers behind the engines can reach the pages at all. The pages second, so the pricing page has a price on it and the integrations page names the PMS products. Structured data third. Then llms.txt and the twins, generated in the build, in an afternoon, and never touched by hand.

A vendor whose evaluators include technical buyers, or whose integration partners build against a public API, has the narrow reason to publish twins. A vendor whose pricing page says “tailored to your portfolio” has a better use for the afternoon. The AI search optimization overview puts those steps in order, and the field notes continue on the blog.