llms.txt: what it is and whether it matters
Updated 2026-07-05
llms.txt is a proposed plain text file placed at the root of a domain. It is meant to give AI systems a short, structured summary of a site: what it is, what its most important pages are, and where to find them. Its proposal positions it as a lightweight companion to robots.txt and sitemap.xml. It targets large language models rather than traditional search crawlers.
The format
The file is plain Markdown served at /llms.txt. A typical file opens with an H1 naming the site, a short blockquote summary, and one or more H2 sections listing links with a brief description each. An "Optional" or "Docs" section pointing at key resource pages is a common example. No required schema exists beyond that loose Markdown structure. No enforcement mechanism exists either: a crawler can read it, ignore it, or never fetch it at all.
A worked example
A minimal, realistic file for a small tool site opens with an H1 carrying the site name and a one line summary of what the tool does. An H2 heading such as "Guides" follows with plain links to the site's cornerstone pages and a short description next to each one. sitemap.digital's own file follows exactly this pattern: name, summary, then links to the homepage and each of the six guides in this resource hub.
Does it actually matter?
The evidence is mixed today. No major AI lab has confirmed that production crawlers like GPTBot or ClaudeBot currently parse llms.txt and change behaviour based on it. Some smaller tools and retrieval pipelines do read it. llms.txt clearly does not replace the fundamentals: a crawler still needs robots.txt access, a real internal link graph, and raw HTML content it can read without running JavaScript. Treat llms.txt as a cheap, forward looking addition. It is not a substitute for those basics.
What sitemap.digital checks
Every scan looks for an llms.txt file at the site root and reports whether it exists. This check sits alongside checks for robots.txt rules covering the major AI bots. Access and structure remain the parts of the score proven to matter today. See the AI readiness score guide for exactly how access is weighted against the other four parts of the score.
Frequently asked questions
Is llms.txt an official web standard?
No. It is a community proposal rather than a standard ratified by a body like the IETF or W3C. No major AI company has publicly confirmed that its crawlers read or act on the file today.
Should I still add an llms.txt file?
Yes. Most sites should. It costs almost nothing to add, does no harm, and keeps you ready if AI systems start consulting it more directly. Treat it as a cheap, low priority addition rather than a must have fix.
Does sitemap.digital check for llms.txt?
Yes. Every scan checks whether an llms.txt file exists at the site root and reports it alongside the standard robots.txt and AI bot access checks.
See how your own site scores against these checks.
Scan your site