Guide

AI crawlability score: how it is calculated

How sitemap.digital scores a page for AI crawlability out of 100: content 50, structure 20, access 10, metadata 10, schema 10.

By Sunny Patel · Updated 2026-07-17

This score measures one thing only: how ready a page is to be crawled and understood by AI search bots such as GPTBot, ChatGPT-User, ClaudeBot, and Google-Extended. It reads raw HTML rather than a rendered browser view. That raw HTML is the same bytes those bots actually download. This is not a corporate AI readiness or AI-adoption-maturity assessment. That separate category of tool measures whether an organisation is set up to adopt AI internally. It is a different topic entirely. If that is what you were looking for, this page will not help and you should look elsewhere instead. If you want to know whether your own website's pages are legible to AI crawlers, keep reading.

Every crawled page that returns a normal 2xx HTML response gets scored out of 100 for AI crawlability across five fixed parts: content worth up to 50 points, structure worth up to 20, access worth up to 10, metadata worth up to 10, and schema worth up to 10. This page publishes the exact method behind that score on purpose. A scoring method you cannot check is not a method you can trust.

This weighting replaced an earlier 30/25/20/15/10 split after an internal validation study compared the score against an independent AI judge on real pages (see the linked study below). The old model gave every reachable page a floor of roughly 35 to 40 points before it had said anything at all, because access paid a flat 25 points on almost every page and metadata barely varied. This version fixes that: access is worth far less and doubles as a gate, content is worth far more and rewards density rather than raw volume, and structure no longer punishes a well written page just for lacking headings.

Content: 0 to 50 points

This part counts the visible words present in a page's raw HTML before any JavaScript runs. It excludes script, style, hidden template content, and now also nav, header, and footer boilerplate, so the count reflects the page's own words rather than furniture repeated on every URL. A page with 450 or more of those honest words scores the full volume share of this part. Fewer words score proportionally less. This is deliberate: an AI crawler that does not execute JavaScript can only read what is already in the HTML it downloads. A page built entirely as a JavaScript shell injects content only after it loads in a browser. It has close to zero words in its raw HTML and scores close to zero here. This holds even when it looks completely normal to a human visitor.

On top of the volume share, a page can earn up to 10 further points for specificity: how many of its words are numbers or proper nouns, the kind of concrete, citable detail an AI judge actually rewards. Two pages with the same word count are not equally useful, and this bonus is what separates a page full of exact facts from one padded with generic filler.

Content also carries the same-shell check. If a page's opening text is shared verbatim by two or more other pages on the site, all of those pages score zero on content, however many words they contain. A page serving another page's body is not content, even when a plain word count would say otherwise.

Access: 0 to 10 points and a gate

A page marked noindex, or a page blocked from every one of the five major AI crawlers tracked (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended), does not just lose access points: the whole score is capped at 20, whatever the other four parts add up to. Reachability is treated as a gate rather than a flat bonus, because a page an AI cannot reach at all cannot be quite ready, no matter how good its content is. Blocking some but not all of the five major bots costs a proportional share of this part's 10 points without triggering the gate.

OpenAI's own crawler documentation confirms GPTBot is disallowed independently from ChatGPT-User and OAI-SearchBot. Each bot gets its own line in robots.txt. Google publishes the same separation for Google-Extended in its list of common crawlers. Blocking Google-Extended stops a site's content from training or grounding Gemini models. It does not affect normal Search ranking. The access part of this score reads exactly those per-bot robots.txt rules and nothing else. It does not confirm a bot actually visited.

Structure: 0 to 20 points

Structure checks four things. Exactly one H1 on the page earns 7 points. At least one H2 or H3 heading earns another 5 points. A healthy ratio of visible text to total markup earns 4 points. A separate 4 points reward substantial prose on its own: a page with 400 or more honest words earns this credit whether or not it has any headings at all. That last point is new on purpose. A long, well argued essay with no headings and no schema used to score near the floor of this part. It no longer does: prose is useful to an AI whether or not it is scaffolded with headings.

Metadata: 0 to 10 points

A title tag earns 2 points for existing and 1 more if it stays within 60 characters. A meta description works the same way: 2 for existing, 1 more if it stays within 160 characters. A canonical tag that is either absent or points back to the page itself earns the final 4. A canonical pointing somewhere else tells crawlers this is not the page to index. It loses those points deliberately. This part's weight was cut from an earlier, larger share because metadata barely varies from page to page and barely predicted real usefulness in testing.

Those length limits are the same ones behind the long title and long description flags, on purpose. An earlier version of this score paid the full points for a title that merely existed, which meant a page could be told its metadata was fine and, in the same report, that its title was too long. A score that contradicts the issue list is worse than no score, so the two now answer to one rule.

Schema: 0 to 10 points

Any structured data on the page at all earns 5 points. A further 5 points get awarded if at least one schema type is content specific: Article, Product, or FAQPage rather than only generic types like WebPage, WebSite, BreadcrumbList, or Organization. Those generic types describe the site rather than the content on the page.

The buckets

The five parts get summed into a total score of up to 100 and grouped into three buckets. 80 and above is strong. 50 to 79 is moderate. Anything below 50 is weak and gets flagged on that page as a low AI score issue. Orphan pages and deep pages use the same flag type. It shows up in the same cross filter across the report.

Why the weights changed

This model replaced an earlier version after an internal validation study compared the score against an independent AI judge across a real sample of pages. The finding: the old model handed every reachable page a floor of roughly 35 to 40 points before it had said anything, because access paid a flat 25 points on nearly every page regardless of content, and metadata barely varied either way. The study also found the score could be fooled by pages that had plenty of raw-HTML words but none of them about the page's own topic, serving a shared shell body across many routes instead. This version answers both problems directly: access is now a small part and a gate rather than a flat bonus, and any page sharing an identical opening with other pages on the same site scores zero on content regardless of its word count.

Why the disambiguation matters

Search interest in the phrase AI readiness is dominated by a different product category: organisational assessments like the kind consultancies run on a company's data infrastructure, governance, and staff skills before an AI rollout. Those tools produce a maturity score for a business. This tool produces a crawlability score for a web page. They share a phrase and nothing else. sitemap.digital publishes its exact weighting for that reason. Anyone comparing the two can see immediately that this score is a structural, technical measure of one page's HTML. It is not a strategic assessment of an organisation.

sitemap.digital's own scan history is still small. No aggregate percentage is published as an authoritative benchmark here. What is published precisely is the rule this tool applies to every page it scores. Your own site's real numbers appear the moment you run a scan.

Frequently asked questions