Learning centre
AI search and technical SEO guides
Practical explanations for the checks in a sitemap.digital scan: site structure, crawler access and visibility in AI search.
Site structure
Find buried pages, weak internal links and crawl paths that make important content hard to reach.
Orphan pages in SEO: what they are and how to fix them
What an orphan page is, how sitemap.digital finds one from real crawl data, and the fastest way to fix it before your next scan.
GuideClick depth in SEO: what it is and why it matters
Click depth and crawl depth in SEO: what each term means, why depth over four hops gets flagged, and how to flatten a site so more of it gets crawled.
GuideInternal links and the internal link graph for AI crawlers
How the internal link graph controls what GPTBot, ClaudeBot, and PerplexityBot can find on your site, and the audit checklist to fix it.
GuideCrawl budget: what it is and who needs to fix it
What crawl budget actually means, which sites need to manage it, how to spot the waste, and the fixes ranked by impact.
GuideSite architecture for SEO and AI search
Flat versus deep structure, hub and spoke linking, and how AI crawlers read your site, with a worked example and the common mistakes.
GuideBest sitemap generators in 2026: XML, visual and HTML
A plain comparison of real XML, visual and WordPress sitemap generators: what each one is for, who it suits, and where to check current pricing.
GuideHow to find orphan pages in Screaming Frog
Steps for finding orphan pages with Screaming Frog Crawl Analysis, what counts as an orphan in each tool, and a free crawl-based alternative.
AI crawler access
Understand robots.txt, llms.txt and the different search, user-fetch and model-training crawlers.
llms.txt: what it is and whether it matters
The llms.txt format explained: what it is meant to do, a real worked example, and an honest, evidence based look at whether it currently changes anything.
GuideAI Crawlers: The Complete List of Major AI Bots (2026)
Every major AI crawler explained by vendor and job: GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot and Google-Extended. Includes a robots.txt example.
GuideClaudeBot: Anthropic's Crawler and How to Control It
ClaudeBot is Anthropic's training crawler. Learn its user agent, how it differs from Claude-SearchBot and Claude-User, and how to block it safely.
GuideGPTBot Explained: OpenAI's Crawlers and How to Control Them
GPTBot, OAI-SearchBot and ChatGPT-User are three separate OpenAI crawlers with different jobs. See the UA tokens and the robots.txt rules that control each one.
GuidePerplexityBot: Perplexity's Crawler and How to Control It
Learn what PerplexityBot is, how to verify it with IP ranges, and how to block it in robots.txt without losing Perplexity's citation traffic.
GuideBytespider: ByteDance's AI crawler and how to block it
Bytespider is ByteDance's AI crawler with no published documentation, verification or IP ranges. Learn what it does and how to block it reliably.
GuideOAI-SearchBot Explained: ChatGPT Search Crawler Guide
OAI-SearchBot builds the index ChatGPT search cites from. See its user agent string, how it differs from GPTBot, and how to check access.
GuideIndexNow explained: setup, limits, and who honours it
What IndexNow does, which search engines actually honour it, how to set up the key file, and how it relates to a sitemap.
AI search visibility
Measure readiness, diagnose invisible content and track whether AI search systems mention your brand.
AI crawlability score: how it is calculated
How sitemap.digital scores a page for AI crawlability out of 100: content 50, structure 20, access 10, metadata 10, schema 10.
GuideWhy your site is not showing up in ChatGPT and AI answers
Your pages are indexed in Google, yet ChatGPT and other AI answers never mention your site. Here is why that gap exists and what to check.
GuideBest AI Search Visibility and Monitoring Tools (2026)
Compare 9 AI search visibility tools spanning free crawlability checks to brand mention trackers like Profound, Otterly and Peec. Honest pricing and limits.
GuideHow to Track Brand Mentions in AI Search (Manual + Tools)
A practical method for tracking whether ChatGPT, Perplexity and Google AI Mode mention your brand. Manual prompts, metrics, tools and a monthly cadence.
GuideHow to get cited by ChatGPT: what actually decides it
The mechanism behind a ChatGPT citation, the factors you can actually control, what does not work, and how to check your own site.