llms.txt

On many websites there are two files not meant for people: robots.txt tells search engines which areas they should not fetch, sitemap.xml lists every address. In concept, llms.txt lines up alongside them. It is meant to tell a language model in a few lines what the website is about and where the important content sits.

The decisive difference belongs at the start: robots.txt and sitemap.xml have been established for decades and are evaluated. llms.txt was published as a proposal in September 2024. No standards body has adopted it, and we know of no public commitment by a major provider to actually read the file. Anyone presenting it as an obligation or an established standard is going beyond what can be substantiated.

Its structure is deliberately kept simple. A heading with the company name, a short paragraph describing in ordinary sentences what is on offer, and below that sections with links to the most important pages, each with an explanatory sentence. It is written in Markdown, a notation for simple text files. No programming is needed for it.

Why it can make sense despite the unclear benefit: the effort is small and the possible harm is nil. And it forces an exercise that is worth doing anyway — putting your own offering into clear, short sentences without advertising vocabulary. Whoever does that for the file regularly notices that the same sentences would work better on the homepage than what is currently there.

One experience we pass on, because it caught us out ourselves: the file is a maintenance point of its own. What is written there stays written there and does not travel along automatically when you change prices, services or languages on the website. In our own file we found details that had long since been corrected elsewhere — in, of all places, the file a model would quote from. Whoever creates it has to fold it into the routine for changes.

As for the benefit we are reserved, but we take the subject behind it seriously. That answers are increasingly assembled by machines is measured, not assumed. Only the route there runs through the content on the page itself — a clear question, a short complete answer, a figure with a source. What counts in that regard is set out under AI Overview and AI visibility.

To draw the line, because the three files are often confused: robots.txt governs access, sitemap.xml lists the addresses, llms.txt describes the content. None replaces another. And none of them compels anything — technically, robots.txt too is a request, which reputable retrievers honour and others do not.

For you that means: if a business wants the file, we create it; it costs next to nothing. We say alongside it that its effect is not substantiated, and we do not present it as a headline service. What we recommend instead is work on the pages themselves — there, the connection between effort and result can be demonstrated.