Guide

llms.txt: what it is and whether it matters

The llms.txt format explained: what it is meant to do, a real worked example, and an honest, evidence based look at whether it currently changes anything.

By Sunny Patel · Updated 2026-07-15

llms.txt is a proposed plain text file placed at the root of a domain. It is meant to give AI systems a short, structured summary of a site: what it is, what its most important pages are, and where to find them. It comes up constantly in AI search and AI visibility discussions, usually framed as a quick win, so it is worth answering both parts of that framing separately: what the file actually does, and whether adding one changes anything measurable today.

The original proposal positions llms.txt as a lightweight companion to robots.txt and sitemap.xml, aimed at large language models rather than traditional search crawlers. The spec itself lives at llmstxt.org and is short enough to read end to end in a few minutes.

The format explained

The file is plain Markdown served at /llms.txt. A typical file opens with an H1 naming the site, a short blockquote summary of what it does, and one or more H2 sections listing links with a brief description next to each one. An "Optional" or "Docs" section pointing at secondary resource pages is a common pattern, used to signal which links matter less if a consumer has limited space to work with. No required schema exists beyond that loose Markdown structure, and no enforcement mechanism exists either. A crawler can read the file, ignore it, or never fetch it at all, and there is no way for a site owner to tell which of those happened.

This is a real difference from robots.txt and sitemap.xml, both of which have a documented history of production crawlers actually consuming them. robots.txt controls access and sitemap.xml aids discovery, and both predate large language models by decades. llms.txt was designed by analogy to those two files, but analogy is not the same as adoption. A site can implement the format perfectly and still have no evidence that anything reads it.

A worked example

sitemap.digital publishes its own llms.txt file at the site root. It is a real, live file rather than an invented sample, and it follows the format above closely: an H1 with the site name, a one line blockquote summary, a "Guides" section linking each cornerstone page with a short description, and an "Optional" section for the homepage.

# sitemap.digital

> sitemap.digital scans a site's crawl structure and reports how ready it is
for AI crawlers and search engines to read, index and cite it.

sitemap.digital crawls a site, builds its internal link graph and scores
every page for AI readiness across content, access, structure, metadata and
schema. It flags orphan pages, pages buried too deep to reach, and access
rules that block major AI bots such as GPTBot, ClaudeBot and PerplexityBot.

## Guides

- [Orphan pages: what they are and how to fix them](https://sitemap.digital/learn/orphan-pages): what an orphan page is and how to fix it.
- [Click depth and discoverability](https://sitemap.digital/learn/click-depth): why pages deeper than four clicks get flagged.
- [llms.txt: what it is and whether it matters](https://sitemap.digital/learn/llms-txt): the format explained plus an honest verdict.

## Optional

- [Homepage](https://sitemap.digital/): run a free scan and see the AI readiness score for any site.

Nothing about that structure is enforced by a schema or a parser. It is a convention that happens to have caught on, which is also why files in the wild vary quite a bit in how many sections and links they include. Some list every page on a site, others stay to a handful of cornerstone pages the way the example above does. Neither approach is wrong, because there is no spec test to fail.

Does it actually matter?

The honest answer is: not much yet. Publishing a file is not the same as anyone consuming it, and that gap is the part most hype pieces skip. A file existing at a known path costs a site owner minutes to write. A crawler actually fetching it, parsing it, and changing what it retrieves as a result is a different claim entirely, and it is the claim that lacks confirmed evidence. llms.txt also does not replace the fundamentals: a crawler still needs robots.txt access, a real internal link graph, and raw HTML content it can read without running JavaScript. Treat llms.txt as a cheap, forward looking addition rather than a fix for a site that fails on those basics.

Independent research that crawled roughly 137,000 domains found that adoption is rising, yet 97 percent of the llms.txt files that exist receive effectively zero real fetch requests from AI crawlers, according to Ahrefs' analysis. sitemap.digital treats existence and use as two separate facts: whether the file is present is something a scan can verify directly, whether it gets used is something no scan, including this one, can currently prove.

What has been said publicly

The public record so far is short and mostly one sided:

• 2024: Jeremy Howard proposes the llms.txt format and publishes the spec at llmstxt.org.
• 2025 into 2026: a growing list of companies, including Anthropic and Cloudflare, publish their own llms.txt files as producers, which signals interest on the publishing side even without confirming that any crawler consumes them.
• 2025 into 2026: Google's own Search Relations team, including Gary Illyes, states publicly that Google does not use llms.txt for Search or AI Overviews and has no plan to, comparing it to the old keywords meta tag: a self declared, unverifiable signal Google abandoned long ago. Google's guidance for site owners lists it by name as a tactic that does not help, reported by Search Engine Land.
• 2026: independent adoption studies, including the Ahrefs analysis cited above, confirm rising publication counts alongside near zero real fetch activity, which is the closest thing to hard usage evidence that exists today.

No source in that timeline shows a major AI lab confirming that a production crawler reads llms.txt and changes its behaviour because of it. That gap is the whole reason for the honest verdict above rather than a hype one.

Should you add one

Yes, with the priority set correctly. Writing a short llms.txt file following the example above takes minutes: name the site, summarise it in one line, and link the pages that actually matter. That effort is small enough that the honest uncertainty over whether anything reads it should not stop you. What it should stop is treating llms.txt as a substitute for the checks that are proven to matter: pages reachable through a real internal link graph, robots.txt rules that do not block major AI bots, and content available in the raw HTML a crawler receives before any JavaScript runs. See the AI readiness score guide for exactly how access is weighted against the other four parts of that score, and fix those first if a scan flags them.

Frequently asked questions