llms.txt

llms.txt is a proposed web standard, introduced by Jeremy Howard in 2024: a markdown file served at a website's root (/llms.txt) that gives AI systems a curated, token-efficient map of the site — what it is, what matters, and where the substantive content lives, with links preferably pointing at markdown versions of each page. Where robots.txt tells crawlers what they may not read and a sitemap enumerates everything indiscriminately, llms.txt tells a language model what it should read, in the format it reads best.

The Problem It Solves

Rendered HTML is a hostile format for a context window: navigation chrome, scripts, and markup typically outweigh content several-fold, and a model burning tokens on boilerplate has fewer left for reasoning. As AI search (ChatGPT, Claude, Perplexity, Gemini) became a meaningful discovery channel, sites gained an incentive to be legible to models — the same incentive that once drove SEO. llms.txt is the legibility layer: a concise briefing plus a curated link list, often accompanied by .md twins of important pages and an expanded llms-full.txt containing full content for single-fetch consumption.

Part of a Larger Pattern

llms.txt completes a trilogy of markdown files at the human–AI boundary: CLAUDE.md briefs an agent about a codebase, SKILL.md briefs an agent about a capability, and llms.txt briefs an agent about a website. All three make the same wager — that plain, versionable markdown is the durable interface between human intent and machine consumption. Adoption is real but contested: thousands of sites publish the files (Anthropic, Cloudflare, and much of the AI tooling ecosystem among them), while major crawler operators haven't formally committed to consuming them — publishing one today is cheap insurance, not guaranteed traffic.

In Practice

Publishing llms.txt is increasingly a platform feature rather than a manual chore — LightCMS, the CMS serving this site, generates it automatically from content. Measuring whether AI systems actually see your brand is the other half of the loop: that's the province of AI-search visibility tools like LLM Optimizer. And because llms.txt files and their .md page twins are ordinary markdown in a repo or CMS, they benefit from the same review-and-check machinery as any other document that machines will read literally — a stale llms.txt confidently misdirects every model that trusts it.