Free assessment

AI readiness assessment for your website

Check whether AI crawlers can read, cite and recommend your website. A free instant assessment covering access, structure, metadata, schema and llms.txt.

By Sunny Patel · Updated 2026-08-23

"AI readiness assessment" usually means something this page does not cover. Most search results use the phrase for an organisational audit. Consultancies use it to measure whether a company is prepared to adopt AI internally. That kind of assessment looks at data maturity, governance, staff skills and change management. This page does not cover that meaning. Search for an enterprise AI readiness framework or speak to a consultancy that runs organisational audits.

This page covers the other meaning. Website AI readiness asks a narrower question. Can AI systems such as ChatGPT, Claude and Perplexity fetch your website, understand it, and cite it when they answer questions? That is a technical property of your site. You can measure it in under a minute and fix most gaps in an afternoon. The rest of this page explains what an AI-ready website looks like, the concrete checks that define readiness, and how to assess your own site.

What an AI-ready website looks like

An AI-ready website is one an AI crawler can fetch, parse and quote without obstacles. Five checks define it in practice. None of them are exotic and most overlap with ordinary technical SEO. The difference is tolerance. Google renders JavaScript, guesses at intent and forgives messy markup. AI crawlers are blunter. A gap that Google papers over can leave a site effectively invisible to AI search.

• AI crawlers are permitted to access your pages
• Your content is present in the raw HTML. It is not assembled by JavaScript
• Pages are structured with clear headings and reachable by internal links
• Metadata describes each page accurately
• Schema and llms.txt give machines an explicit summary of what the site is

Check 1: crawler access

The first check is the bluntest. AI companies crawl the web with named bots. OpenAI documents GPTBot, Anthropic documents ClaudeBot, and Perplexity documents PerplexityBot. Your robots.txt file decides whether each of these is allowed in. Sites block them without realising it all the time, usually through a blanket disallow rule, a block list copied from a template, or a CDN firewall rule that filters anything labelled as a bot.

A robots.txt block means a compliant AI crawler will not fetch your pages at all. Nothing else on this list matters until access is open. Open yourdomain.com/robots.txt and search for the major AI user agents by name. A robots.txt check has one limit. A robots.txt check tells you what crawlers are permitted to do. It does not tell you what they actually do. sitemap.digital reads permissions. It does not observe crawler traffic.

Check 2: content that survives without JavaScript

The second check is whether your content exists in the HTML your server sends. AI crawlers generally read the raw response and take what is there. Client-rendered pages that arrive as an empty shell show an AI crawler a navigation bar and little else. Single page applications built in React, Vue or similar frameworks are the usual casualties when they render entirely on the client.

The test takes thirty seconds. View the page source in your browser and search for a sentence from your main content. Treat a missing sentence in the source as proof AI systems cannot read that page. Server side rendering or static generation fixes this at the framework level. Most modern frameworks support both.

Check 3: structure, links and click depth

AI systems split pages into chunks and retrieve the chunks that answer a question. Clear structure makes that retrieval work. Each page needs one h1 that states the topic, headings that honestly describe the sections beneath them, and paragraphs that make sense when read in isolation. A wall of text under a vague heading retrieves poorly even when the information inside it is good.

Site level structure matters just as much. Pages with no internal link pointing to them are called orphan pages. Orphan pages are hard for any crawler to discover. Pages buried four or five clicks from the homepage get crawled less and cited less. Broken links waste crawl effort and erode trust in the rest of the site. An assessment should surface all three. None of them are visible from casually browsing your own site.

Check 4: metadata and schema

Titles and meta descriptions are the oldest signals on this list and they still do work. An AI system deciding whether a page answers a question reads the title first. A title that names the topic plainly beats a clever one. Canonical tags matter too. Duplicate URLs split whatever authority a page has.

Schema markup in JSON-LD goes further by making explicit, typed statements about a page. It declares that this is an article, this is the author, this is an FAQ with these questions. Machines do not have to infer any of it. You do not need schema on every page. Start with Organization and WebSite markup site wide. Add Article or FAQ markup next where the page type fits.

Check 5: llms.txt

llms.txt is a proposed standard for a markdown file at your site root that gives language models a curated summary of the site and links to its most important content. The proposal is documented at llmstxt.org. One caveat applies here. No major AI provider has publicly committed to reading llms.txt. Treat it as low cost insurance rather than a lever. It takes minutes to add. It cannot hurt. Adoption growth leaves you already covered. The scanner checks for it and can generate one for you.

Assess your site in under a minute

Run every check above by hand or let a scanner do it in one pass. sitemap.digital is a free instant scanner with no signup. It crawls your site, scores every page from 0 to 100 for AI crawlability, checks robots.txt access for GPTBot, ClaudeBot, PerplexityBot and other major AI crawlers, checks for llms.txt, and finds orphan pages, broken links and click depth problems. The free scan covers 20 pages. A free email unlock extends it to 50. Small sites are crawled to completion. The score weights content at 50 points, structure at 20, and access, metadata and schema at 10 each. The AI readiness score guide explains the full method. This page will not repeat it.

Fix issues in this order once results come back. Fix access first. A blocked crawler makes every other fix pointless. Fix JavaScript legibility second. Invisible content cannot be cited. Then structure and internal links, then metadata and schema, then llms.txt. The scanner generates done-for-you fix files for the mechanical parts, including an llms.txt, robots.txt snippets, schema stubs and redirect maps. Most of the remaining work is judgement rather than typing.

Frequently asked questions