The state of AI crawlability
This is live aggregate data from every site scanned with sitemap.digital, updated continuously as new scans complete. It is not a survey and not a random sample of the web: it reflects the sites people chose to scan with this tool. Every figure below is an aggregate across the sample, never a single domain.
Two of these figures rest on different samples, and the cards say which. Anything read from robots.txt, llms.txt or the sitemap counts every completed scan, because one fetch answers it. Anything derived from the crawled pages themselves, orphan pages and broken links above all, counts only scans that finished without hitting their page cap. A capped crawl takes the first pages the sitemap happens to list, so a page can look orphaned purely because whatever linked to it was never fetched. Reporting that as an orphan rate would describe our crawl order rather than the web.
Loading the latest figures…
Why this matters
AI systems such as ChatGPT, Claude, and Perplexity read the web the same way any crawler does: they request raw HTML, they follow links, and they check robots.txt before doing either. A site can look complete to a human visitor in a browser and still be close to invisible to a crawler that never runs its JavaScript, never finds a link pointing at a page, or is told to stay out entirely in robots.txt. The figures above are the same checks sitemap.digital runs on every individual scan, rolled up into one aggregate picture across the sample.
None of this is a ranking signal or a traffic guarantee. A blocked bot, a missing llms.txt file, or an orphan page describes a structural gap, not a cause of lost visibility on its own. Treat these numbers as a baseline for how prepared the average scanned site is, then check your own site’s real numbers with a scan.
See where your own site stands
These are averages across every scanned site. Run your own scan to see your real numbers, page by page.