llms.txt

llms.txt is a proposed web standard, introduced by Jeremy Howard in 2024: a markdown file served at a website's root (/llms.txt) that gives AI systems a curated, token-efficient map of the site — what it is, what matters, and where the substantive content lives, with links preferably pointing at markdown versions of each page. Where robots.txt tells crawlers what they may not read and a sitemap enumerates everything indiscriminately, llms.txt tells a language model what it should read, in the format it reads best.

The Problem It Solves

Rendered HTML is a hostile format for a context window: navigation chrome, scripts, and markup typically outweigh content several-fold, and a model burning tokens on boilerplate has fewer left for reasoning. As AI search (ChatGPT, Claude, Perplexity, Gemini) became a meaningful discovery channel, sites gained an incentive to be legible to models — the same incentive that once drove SEO. llms.txt is the legibility layer: a concise briefing plus a curated link list, often accompanied by .md twins of important pages and an expanded llms-full.txt containing full content for single-fetch consumption.

Part of a Larger Pattern

llms.txt completes a trilogy of markdown files at the human–AI boundary: CLAUDE.md briefs an agent about a codebase, SKILL.md briefs an agent about a capability, and llms.txt briefs an agent about a website. All three make the same wager — that plain, versionable markdown is the durable interface between human intent and machine consumption. Adoption is real but contested: thousands of sites publish the files (Anthropic, Cloudflare, and much of the AI tooling ecosystem among them), while major crawler operators haven't formally committed to consuming them — publishing one today is cheap insurance, not guaranteed traffic.

Who Actually Reads It

The first large-scale usage data arrived in 2026 and it narrows the claim considerably. Ahrefs (June 2026) examined May 2026 server logs for 137,210 domains and found that 97% of llms.txt files received zero requests. AI retrieval bots, the crawlers that feed AI search answers, accounted for only 1.1% of the requests that did occur. The largest AI category was coding agents and agent infrastructure at 10.5%. Ahrefs also reported that "zero AI bots 'go looking' for llms.txt files that don't exist," which means the file is fetched when something points an agent to it, not discovered by convention. As a lever for generative engine optimization, the evidence so far does not support llms.txt: the systems that decide AI search citations are largely not reading it.

The case is stronger where the readers are agents rather than search engines. Mintlify, a documentation host reporting on its own customers (July 2026, vendor data), said agents made up 66% of traffic to the developer docs it hosts, 213 million agent requests against 105 million human page loads, up from 15.2% at the start of 2026, and that adding llms.txt cut agent 404 errors by about 90% in its benchmark. Developer documentation consumed by AI coding agents is therefore the setting where llms.txt has measurable users today, and where it competes and combines with MCP servers as a way to hand an agent accurate context.

In Practice

Publishing llms.txt is increasingly a platform feature rather than a manual chore — LightCMS, the CMS serving this site, generates it automatically from content. Measuring whether AI systems actually see your brand is the other half of the loop: that's the province of AI-search visibility tools like LLM Optimizer. And because llms.txt files and their .md page twins are ordinary markdown in a repo or CMS, they benefit from the same review-and-check machinery as any other document that machines will read literally — a stale llms.txt confidently misdirects every model that trusts it.