Guide

Internal links and the internal link graph for AI crawlers

How the internal link graph controls what GPTBot, ClaudeBot, and PerplexityBot can find on your site, and the audit checklist to fix it.

By Sunny Patel · Updated 2026-07-15

Internal link structure, not backlinks, is what lets AI crawlers find your pages. Your internal link graph, the full set of anchor tags connecting every page on your site to every other page, is often the more decisive factor for AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot. A crawler moves through the site by following the links it finds in the raw HTML once it lands anywhere on your domain. It moves page by page, the same way a visitor clicking through your navigation would, and the shape of that graph, not the quality of any single page, determines what gets read at all.

Most SEO advice about internal linking is written for Google, a crawler with decades of accumulated URL history to fall back on when your own linking falls short. AI crawlers built in the last few years mostly do not have that fallback. Get the internal link graph wrong and a whole section of a site can sit fully published, fully readable, and completely invisible to a crawler that only ever follows links.

Internal links as the literal crawl path

An AI crawler has no map of your entire site in advance. It builds that map one link at a time, starting from wherever it first arrived: often your homepage or a URL it picked up from a sitemap or an external mention. Every anchor tag with a real href attribute gives the crawler a step it can take. Every important page with no link pointing to it becomes a dead end the crawler never reaches. Content quality does not change that: a page can be the single best answer to a question on the entire internet and still never get read if nothing links to it.

This is why the internal link graph matters as a structural object, not just as a collection of individual links. A page two clicks from the homepage, reachable from a heavily linked hub page, gets crawled far more reliably than the same content sitting six clicks deep behind a pagination control. The graph shape decides the odds of a page being found long before anything about its content comes into play.

Why this matters more for LLM bots than for Google

Google has spent over two decades building a URL database from sitemaps, backlinks, historical crawls, and direct submissions through Search Console. That depth of history gives Google many independent ways to stumble onto a page even when its internal linking is poor. Newer AI crawlers generally lack that depth of history on most domains, which pushes them to lean more heavily on a straightforward, link following crawl from whatever entry point they have.

Google itself draws the same distinction for its own AI crawler. Google-Extended, the token that controls whether a site's content can train Gemini models, does not run its own separate crawl. Google's documentation states it plainly: “Google-Extended doesn’t have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings”. In other words, Google's AI crawling rides on Googlebot's decades-deep discovery infrastructure rather than starting from scratch. Newer AI-only crawlers built by companies without that legacy search index do not have an equivalent shortcut. For them, the internal link graph is closer to the whole discovery mechanism rather than one signal among many.

What breaks the crawl path

Links that only work through JavaScript click handlers with no underlying href can turn invisible to a crawler that reads raw HTML and does not execute scripts. Images used as buttons with no accompanying text link cause the same problem. Navigation rendered entirely on the client after the page loads breaks the same way, and so does a “load more” button that only appends new links after a click event fires. A page can look perfectly linked to a person browsing in a normal browser and still sit functionally orphaned from a crawler's point of view, because the two are reading completely different versions of the same URL.

GPTBot vs ClaudeBot vs PerplexityBot: what is actually documented

sitemap.digital does not observe real crawler traffic on your site: no outside tool can, since none of these operators publish public per-request logs. What a scan can tell you is what your robots.txt currently permits for each of these user agents and how well your internal link graph would support a link-following crawl if one of them arrived. The rest below is drawn from each operator's own published documentation, not from observed behaviour.

GPTBot (OpenAI)

OpenAI's own documentation states that GPTBot is used to crawl content that may be used in training OpenAI's generative AI foundation models, and that disallowing GPTBot in robots.txt signals a site's content should not be used for that training. The published documentation does not detail GPTBot's specific page-discovery mechanism beyond confirming it operates as a standard web crawler that checks robots.txt, adding a marker to its user agent string on that specific request so site owners can distinguish it from other fetches.

ClaudeBot (Anthropic)

Anthropic's published support documentation describes ClaudeBot as helping enhance the utility and safety of Anthropic's generative AI models by collecting web content that could potentially contribute to their training. Anthropic states its crawlers respect industry-standard robots.txt directives and support a crawl-delay extension for controlling request frequency, and gives the exact rule to block it: a `User-agent: ClaudeBot` block with `Disallow: /`. The documentation notes blocking by IP address is unreliable, since that can also stop the crawler from reading the robots.txt file that would otherwise tell it to stay away.

PerplexityBot (Perplexity)

Perplexity draws a distinction worth flagging directly, because it cuts against the assumption that all three bots do the same job. Perplexity's own documentation states that PerplexityBot is designed to surface and link websites in search results on Perplexity and is not used to crawl content for AI foundation models. It is a search-indexing crawler, comparable in purpose to Googlebot, rather than a training-data crawler like GPTBot or ClaudeBot. Perplexity documents a separate user agent, Perplexity-User, that fetches a page live in response to a specific user question and can behave differently around robots.txt because it is acting on an explicit request rather than crawling proactively. For the purpose of this guide, the ordinary internal link graph rules apply to PerplexityBot's proactive crawling the same way they apply to GPTBot and ClaudeBot.

The practical takeaway is that these three crawlers are not interchangeable, but the fix is the same regardless of which one you care about most. None of the operator documentation above claims to publish a map of exactly how each crawler prioritises which links to follow first, how deep it goes, or how it revisits a domain over time. What each operator does confirm is standard robots.txt compliance and a stated purpose. Treat the internal link graph as the shared surface all three read from, and build for that, rather than trying to reverse engineer a different crawl strategy for each one.

Step-by-step internal link graph audit checklist

Run these checks against a scan's site structure section in order. Each step uses a field the scan already calculates, so none of this needs a separate tool.

1. Start a scan from your real homepage, not a subpage, so click depth is measured from the same entry point a crawler would actually use.
2. Open the orphan list first. Every page on it has an inbound link count of zero. Decide for each one whether it deserves a link in from somewhere already reachable, or whether it should be retired or redirected instead.
3. Sort pages by click depth. Anything beyond three or four hops from the homepage is competing for a shrinking share of crawl attention. Pull your highest-value deep pages closer by linking to them from a hub page nearer the top of the graph.
4. Check the broken internal edges list. Each entry is a link on your own site pointing at a URL that errored during the crawl. Repoint each one to the correct live URL or remove the link entirely.
5. Cross-check your highest-priority content, the pages you most want an AI system to read and cite, against both the orphan list and the depth ranking. A page with strong content sitting deep or unlinked is the single highest-leverage fix available from one scan.
6. Make the link changes, then rescan the same domain. The diff view shows exactly which pages moved off the orphan list, which pages got shallower, and which broken edges cleared, so you can confirm the fix worked instead of assuming it did.

Worked example: what scan output looks like

The snippet below is an illustrative example built to show the shape of the data, not a result pulled from a real scan. A live scan returns this same structure, per-page inDegree and depth plus a list of edges and broken edges, at whatever scale the site actually has.

{
  "scan": "example.com",
  "pages": [
    { "url": "/", "inDegree": 14, "outDegree": 22, "depth": 0, "orphan": false },
    { "url": "/blog/", "inDegree": 9, "outDegree": 40, "depth": 1, "orphan": false },
    { "url": "/blog/2019-old-post", "inDegree": 0, "outDegree": 3, "depth": null, "orphan": true },
    { "url": "/pricing", "inDegree": 1, "outDegree": 5, "depth": 2, "orphan": false }
  ],
  "edges": [
    { "from": "/", "to": "/blog/" },
    { "from": "/", "to": "/pricing" },
    { "from": "/blog/", "to": "/pricing" },
    { "from": "/blog/", "to": "/contact-us-old" }
  ],
  "brokenEdges": [
    { "from": "/blog/", "to": "/contact-us-old", "status": 404 }
  ]
}

Reading it: the homepage sits at depth 0 with 14 inbound links from elsewhere on the site. The blog index is one hop out and well linked. The old blog post has an inDegree of zero and a null depth, meaning no crawled page links to it at all, so it gets flagged as an orphan regardless of how good the post itself is. The edge from the blog index to a contact page that no longer exists shows up separately in `brokenEdges` with the status code the crawl received, distinct from the orphan flag because the fix here is to repoint or delete that one link rather than add a new inbound link somewhere.

None of this requires guessing at what a crawler might do. The graph a scan produces is the same graph any link-following bot would build for itself while reading your site, so fixing what it flags directly improves what GPTBot, ClaudeBot, and PerplexityBot can each reach on their own next visit, whenever that happens to be.

Frequently asked questions