Guide

Crawl budget: what it is and who needs to fix it

What crawl budget actually means, which sites need to manage it, how to spot the waste, and the fixes ranked by impact.

By Sunny Patel · Updated 2026-09-20

Crawl budget is the number of URLs on your site a search engine is willing and able to fetch inside a given window. Google names two components that set that number: the crawl capacity limit, which is how many simultaneous connections and how much delay Googlebot uses so it does not overload your server, and crawl demand, which is how much Google actually wants to fetch your pages based on their perceived value and freshness. A fast, reliable server raises the first number. Popular, frequently updated content raises the second. Neither number is fixed. Both move as your site and your hosting change.

The two limits interact in a specific order. A generous crawl capacity limit does nothing if crawl demand is low, because the engine simply will not choose to spend the allowance you have available. A high crawl demand does nothing useful if your server keeps timing out or returning errors, because the crawler backs off the capacity limit automatically to protect your site. Fixing crawl budget usually means working on demand first: giving a crawler clear reasons to keep revisiting, through fresh content and a clean internal link graph that gets new and updated pages found quickly, before touching server-side rate settings that most sites never need to adjust at all.

Who actually needs to worry about this

Most sites do not have a crawl budget problem worth solving. A site with a few hundred pages and a stable host gets crawled in full within a normal window no matter how tidy its structure is. Google's own guidance on managing crawl budget for large sites states this plainly: it applies to sites in the range of many thousands of URLs, not a brochure site with a dozen service pages and a blog.

Two site profiles cross that line in practice. The first is genuinely large: ecommerce catalogues, marketplaces, and publishers running into the tens or hundreds of thousands of pages, where a crawler has to make real choices about what it can cover in a visit. The second is churny instead of large: a site that generates, deletes, or moves URLs constantly, such as a job board, a listings site, or a platform with heavy faceted navigation. A churny site with only ten thousand pages can waste more crawl budget than a stable site ten times its size, because so much of every visit goes towards URLs that no longer matter by the time they get fetched.

How to see where the budget is going

Crawl budget waste is invisible until you go looking for it in the right place. Five sources tend to surface the same underlying problems from different angles.

• Server log files. The only direct record of what a crawler actually requested, when, and what status code it received. Log analysis shows real behaviour instead of an estimate, and it is the only source that can confirm a crawler is repeatedly hitting a URL nobody expected it to touch.
• Google Search Console's crawl stats report. A summary view of total crawl requests over time, broken down by response code, file type, and purpose. A sudden spike in 404 or 5xx responses, or a plateau in total requests despite a growing site, both point at a demand or rate problem worth investigating further in the logs.
• Orphan and deep pages. A page with no inbound internal link, or one buried many hops from the homepage, competes poorly for whatever budget a crawler brings to a visit. The orphan pages guide covers how these accumulate and why they starve of both crawl attention and link equity at the same time.
• Redirect chains. Every hop in a chain is a separate request before the crawler reaches a page it can actually index. A three hop chain costs three requests for the outcome a single 200 response would have delivered.
• Parameter URLs and soft 404s. A filter, sort order, or tracking parameter can multiply one real page into dozens of near duplicate URLs, each one a separate crawl request for content the engine has already seen. A soft 404, a page that returns a 200 status but shows an empty result or error message to a visitor, wastes a request in a different way: the crawler treats it as valid content worth indexing when it is not.

Fixes ranked by impact

Not every fix below is worth the same effort on every site. This order reflects the typical ratio of impact to work, starting with the changes that tend to pay off fastest.

1. Collapse redirect chains to a single hop. Point every redirect straight at its final destination. This is usually a bulk find-and-replace against a list rather than a page-by-page fix, and it removes wasted requests immediately.
2. Link every valuable orphan page in from a page that is already reachable.One contextual link often does the whole job. Run the orphan page checker to find which pages currently have zero inbound links before deciding where to add one.
3. Consolidate parameter URLs. A canonical tag pointing every parameter variant at the clean URL, combined with a consistent internal linking pattern that never links to the parameter version in the first place, stops the duplication at the source instead of asking the crawler to sort it out.
4. Noindex or remove soft 404s. A thin or empty page that currently returns 200 should either return a genuine 404 or 410, or carry a noindex tag if the URL needs to stay live for other reasons. Either change stops the page competing for crawl attention it cannot earn.
5. Flatten your deepest sections. Adding hub links so your most important pages sit within three or four hops of the homepage is slower to execute than the fixes above but compounds every time a new page gets added under that same hub going forward.

Run a scan to see your own orphan count, depth distribution, and redirect-offsite flags before deciding which of these to tackle first. The full site audit builds on the same crawl to turn each finding into a ranked action list instead of a raw number.

Frequently asked questions