# llms.txt

> llms.txt is a proposed standard: a markdown file at a website's root that gives AI systems a curated, token-efficient map of the site's content. How it works, how it relates to robots.txt and sitemaps, and why it completes markdown's takeover of the human-AI interface.

Source: https://metavert.io/llms-txt  
Published: 2026-07-04  
Updated: 2026-10-07

**llms.txt** is a proposed web standard, introduced by Jeremy Howard in 2024: a [markdown](https://metavert.io/markdown) file served at a website's root (`/llms.txt`) that gives AI systems a curated, token-efficient map of the site — what it is, what matters, and where the substantive content lives, with links preferably pointing at markdown versions of each page. Where `robots.txt` tells crawlers what they *may not* read and a sitemap enumerates everything indiscriminately, llms.txt tells a language model what it *should* read, in the format it reads best.

### The Problem It Solves

Rendered HTML is a hostile format for a context window: navigation chrome, scripts, and markup typically outweigh content several-fold, and a model burning tokens on boilerplate has fewer left for reasoning. As AI search (ChatGPT, Claude, Perplexity, Gemini) became a meaningful discovery channel, sites gained an incentive to be *legible to models* — the same incentive that once drove SEO. llms.txt is the legibility layer: a concise briefing plus a curated link list, often accompanied by `.md` twins of important pages and an expanded `llms-full.txt` containing full content for single-fetch consumption.

### Part of a Larger Pattern

llms.txt completes a trilogy of markdown files at the human–AI boundary: [CLAUDE.md](https://metavert.io/CLAUDE.md) briefs an agent about a *codebase*, [SKILL.md](https://metavert.io/SKILL.md) briefs an agent about a *capability*, and llms.txt briefs an agent about a *website*. All three make the same wager — that plain, versionable markdown is the durable interface between human intent and machine consumption. Adoption is real but contested: thousands of sites publish the files (Anthropic, Cloudflare, and much of the AI tooling ecosystem among them), while major crawler operators haven't formally committed to consuming them — publishing one today is cheap insurance, not guaranteed traffic.

### Who Actually Reads It

The first large-scale usage data arrived in 2026 and it narrows the claim considerably. Ahrefs (June 2026) examined May 2026 server logs for 137,210 domains and found that 97% of llms.txt files received zero requests. AI retrieval bots, the crawlers that feed AI search answers, accounted for only 1.1% of the requests that did occur. The largest AI category was coding agents and agent infrastructure at 10.5%. Ahrefs also reported that "zero AI bots 'go looking' for llms.txt files that don't exist," which means the file is fetched when something points an agent to it, not discovered by convention. As a lever for [generative engine optimization](https://metavert.io/generative-engine-optimization), the evidence so far does not support llms.txt: the systems that decide AI search citations are largely not reading it.

The case is stronger where the readers are agents rather than search engines. Mintlify, a documentation host reporting on its own customers (July 2026, vendor data), said agents made up 66% of traffic to the developer docs it hosts, 213 million agent requests against 105 million human page loads, up from 15.2% at the start of 2026, and that adding llms.txt cut agent 404 errors by about 90% in its benchmark. Developer documentation consumed by [AI coding agents](https://metavert.io/ai-coding-agents) is therefore the setting where llms.txt has measurable users today, and where it competes and combines with [MCP servers](https://metavert.io/compare/llms-txt-vs-mcp) as a way to hand an agent accurate context.

### In Practice

Publishing llms.txt is increasingly a platform feature rather than a manual chore — [LightCMS](https://metavert.io/lightcms), the CMS serving this site, generates it automatically from content. Measuring whether AI systems actually see your brand is the other half of the loop: that's the province of AI-search visibility tools like [LLM Optimizer](https://metavert.io/llmopt). And because llms.txt files and their `.md` page twins are ordinary markdown in a repo or CMS, they benefit from the same [review-and-check machinery](https://metavert.io/docs-as-code) as any other document that machines will read literally — a stale llms.txt confidently misdirects every model that trusts it.

## Related Topics

- [Markdown](https://metavert.io/markdown) — The format llms.txt standardizes on
- [CLAUDE.md](https://metavert.io/CLAUDE.md) — The codebase briefing in the same trilogy
- [SKILL.md](https://metavert.io/SKILL.md) — The capability briefing in the same trilogy
- [LightCMS](https://metavert.io/lightcms) — Generates llms.txt automatically
- [LLM Optimizer](https://metavert.io/llmopt) — Measures your AI-search visibility
- [Docs as Code](https://metavert.io/docs-as-code) — Review machinery for machine-read documents
- [Agentic Web](https://metavert.io/agentic-web-concept) — The broader shift llms.txt serves
- [llms.txt vs robots.txt](https://metavert.io/compare/llms-txt-vs-robots-txt) — A reading guide versus an access policy
- [AI Crawlers](https://metavert.io/ai-crawlers) — The bots that do, and mostly do not, request the file

## Further Reading

- [The llms.txt proposal](https://llmstxt.org/) — Jeremy Howard / Answer.AI
- [llms-txt specification repository](https://github.com/AnswerDotAI/llms-txt) — GitHub
- [llms.txt directory](https://directory.llmstxt.cloud/) — sites publishing the standard
- [llms.txt study: who requests the file across 137,210 domains](https://ahrefs.com/blog/llmstxt-study/) — Ahrefs, June 2026
- [State of Docs Traffic](https://www.mintlify.com/blog/state-of-docs-traffic) — Mintlify, July 2026