What is llms.txt?
LLMs.txt is a proposed file format, published as plain markdown at a website's root address, that gives AI systems a short, curated overview of what a site contains and links to its most important pages, written specifically for a language model to read rather than for a human browsing a page. The idea behind it is straightforward: a web page built for people carries navigation menus, ads, scripts and layout markup that waste an AI system's limited attention when it is trying to extract the actual content, and a clean, purpose-built summary file removes that overhead.
The file was authored by Jeremy Howard, a well-known figure in the AI and open-source community, and first published in September 2024 as a community proposal rather than an official standard. Its structure is deliberately simple: an H1 heading with the project or company name, a short blockquote summary, and a series of markdown links grouped under H2 headings pointing to more detailed documentation or content elsewhere on the site.
The comparison people reach for immediately is robots.txt, and it is useful up to a point. Both are small, plain-text files sitting at a predictable web address. Where they diverge matters more than where they overlap: robots.txt is a set of access rules that well-behaved crawlers are expected to follow, telling them what they may and may not fetch, while llms.txt carries no such authority. It is closer to a curated welcome note left for a visitor than to a locked door, informational rather than enforceable, which is the single most important thing to understand before deciding how much effort it deserves.
Why llms.txt matters
Three points explain why this has become a live question for businesses investing in AI visibility, alongside a clear caveat.
- It is modelled on a format businesses already understand. The comparison to robots.txt makes the concept easy to explain internally, a small text file at the site root that speaks to automated systems, even though the two files serve genuinely different purposes, one about permissions, the other about content guidance.
- Adoption is real but still a minority practice. A large-scale study of several hundred thousand domains found roughly one in ten already publishing an llms.txt file, and the list of adopters includes recognisable technology companies, which signals the practice is more than a fringe experiment without yet being close to universal.
- The major AI crawlers mostly do not request it yet. This is the caveat worth stating plainly rather than glossing over: crawlers from OpenAI, Google and Anthropic do not currently fetch llms.txt files in meaningful volume, which means publishing one today is a forward-looking bet on where AI crawling behaviour is heading, not a lever with an immediate, measurable payoff.
The honest framing is that llms.txt sits somewhere between a genuinely useful practice and a speculative one, worth doing because it is cheap and because the direction of travel favours it, not because it currently guarantees a visibility gain.
How it works
Four elements define the file as currently proposed.
Location and format. A plain markdown file placed at the site root, reachable at a predictable web address, readable by both AI systems and, if someone happens to look, by a person opening it directly.
A required structure. An H1 with the site or project name, a blockquote with a short summary, optional additional context, and H2-delimited sections listing markdown links with brief descriptions pointing to more detailed pages.
Curated, not exhaustive, content. The file is meant to highlight what actually matters, key pages, core documentation, essential context, rather than mirror a full sitemap, since the entire point is to save an AI system from sorting through everything itself.
No enforcement mechanism. Unlike robots.txt, which crawlers are expected to respect as a set of rules, llms.txt is purely informational. Nothing compels an AI system to read it, use it, or treat its contents as authoritative, which is the central difference from access-control files. Platforms that already generate one automatically, including Mintlify and GitBook, tend to do so from existing documentation structure, which is a useful hint for how to think about the file even when writing it by hand, as a distillation of structure that already exists rather than a new piece of content invented from scratch.
Implementation
The order below reflects how to add an llms.txt file without overinvesting in a format still finding its footing.
- Write the summary as if explaining the business to a new hire in two sentences. The blockquote at the top carries the most weight, since it is the part most likely to actually get read and used if the file is picked up at all.
- List only the pages that genuinely matter. Core service or product pages, key definitional content, essential documentation, not every page on the site, since curation is the entire value proposition of the format.
- Keep the file at the site root, in plain markdown. No custom formatting, no non-standard syntax, since the point is machine readability by the widest possible range of tools.
- Treat it as a low-cost addition, not a project. Given that major crawlers do not yet request it broadly, spending more than a few hours on the first version is difficult to justify against other technical SEO or AI-visibility work.
- Update it when the site's core content actually changes. A stale llms.txt pointing to outdated pages is worse than none, since it actively misguides any system that does read it.
- Monitor server logs for AI crawler requests to the file over time. As adoption among AI crawlers evolves, log data is the most reliable way to know whether the investment is starting to pay off for a given site.
What it costs
The direct cost is close to zero, which is exactly what makes llms.txt worth doing even while its real-world payoff is unproven. Writing a first version is typically an hour or two of work for someone who already knows the site's content well, no development budget or specialised tooling required, and platforms including Mintlify and GitBook already generate one automatically as part of their documentation output.
The real cost, as with several lightweight technical SEO practices, is upkeep rather than creation. A file that accurately reflects the site on day one and is never revisited again slowly becomes noise rather than signal, and most businesses fold that maintenance into an existing technical SEO or content review cycle rather than treating it as a separate line item.
There is a smaller opportunity cost worth naming too, not financial but attentional. Time spent polishing an llms.txt file is time not spent on the technical SEO fundamentals that measurably affect both traditional search rankings and AI citation rates today, structured data, clean semantic HTML, clear answer-first content. For a small team with limited hours, sequencing matters more than the file itself, and llms.txt belongs after those fundamentals are already in reasonable shape, not before them.
Conclusion
LLMs.txt is a cheap, low-risk bet on a direction the AI crawling ecosystem may or may not fully move toward, not yet a proven lever for AI visibility. The honest case for adopting it today rests on its near-zero cost and the fact that a growing, credible set of technology companies already publish one, not on any current guarantee that major AI systems will read and act on it.
The businesses getting the most reasonable value from it are the ones treating it as a small, low-effort addition to an already solid technical foundation, structured content, clean HTML, clear definitions, rather than as a substitute for that foundation or a guaranteed shortcut to AI citations.
