Crawl Budget

Crawl budget is the number of pages a search engine fetches on a website within a given period. Google sets it from two factors: how much the server can deliver without slowing down, called crawl capacity, and how much interest Google has in the content, measured on popularity and update frequency, called crawl demand. The budget is never published as a number, though the activity behind it shows up in Google Search Console's crawl stats report.

For most sites with a few hundred or a few thousand pages, Google crawls plentifully and the budget is no bottleneck. It turns into a real constraint on large sites: shops with faceted navigation, where filters and parameters can generate millions of URL variants, or media archives spanning many years. When the budget goes on endless filter combinations, redirect chains and duplicate variants, the important pages are reached less often and new content waits longer for indexing.

Server speed enters the calculation directly, because crawl capacity is measured in what the server can deliver, so faster response times mean more pages fetched at the same load. Robots.txt, the sitemap and the internal link structure are the levers that decide where crawlers spend their visits. AI crawlers such as GPTBot and PerplexityBot do not count against Google's crawl budget, but they draw on the same server. Where the budget actually goes on a large site is unpacked in the guide to crawling and indexing.