Crawl budget (how much Google crawls your site)
Crawl budget is the set of URLs Google can and wants to crawl on a site. Google's documentation defines it through two elements: the crawl capacity limit, which is how much your server can take without slowing down, and crawl demand, which is how much Google wants to recrawl your URLs. It matters for sites with very many pages or very frequent changes; a small site that gets crawled the day it publishes does not need to manage it.
Who should care
Google's guide on optimizing crawl budget opens with a filter. If your pages get crawled the same day they are published, you do not need the guide; an up-to-date sitemap and a regular look at the Page indexing report are enough. The guide is aimed at:
- large sites with over a million unique pages whose content changes moderately often, about once a week,
- medium or larger sites with over 10,000 unique pages whose content changes very rapidly, daily,
- sites with a large share of URLs classified in Search Console as Discovered - currently not indexed.
Google calls these numbers rough estimates, not exact thresholds.
How Google defines it
According to the crawl budget documentation, each site, meaning each unique hostname, gets a limited amount of crawling time and resources. Two elements set the budget.
Crawl capacity limit, also called hostload, limits the total time your server spends holding connections open for Google, counting both parallel connections and their duration. Every site starts with the same conservative default. The limit goes up when the site responds consistently and response times, including latency and time to first byte, stay stable or improve. It goes down when the site slows down, returns 5xx errors or sends rate-limiting signals such as HTTP 429.
Crawl demand is how much Google wants to crawl. For Googlebot it depends on site size, update frequency, page quality and relevance compared with other sites. The factors you can influence are perceived inventory (Google tries to crawl all URLs it knows about, so duplicates and unwanted URLs waste time), popularity (popular URLs are crawled more often) and staleness (Google wants to recrawl documents often enough to pick up changes).
Together, the documentation says, crawl budget is the set of URLs Google can and wants to crawl. Even if the capacity limit is not reached, low demand means less crawling. The capacity limit is shared across all Google crawlers, so heavy AdsBot crawling reduces what is left for Googlebot.
What wastes it
Google's crawl budget guide and its page on reducing the crawl rate name the usual culprits:
| Waste | Why it costs crawls |
|---|---|
| Duplicate content on several URLs | Each copy is a separate crawl of the same text |
| Faceted navigation and sorting parameters | Endless combinations of filters create URLs Google tries to cover |
| Calendars and other infinite URL spaces | Every date is a new URL with little content |
| Soft 404 pages | Google says they keep being crawled and waste budget |
| Long redirect chains | Each hop is a request; Google says chains hurt crawling |
| Noindex used instead of robots.txt | The page is still requested, then dropped |
Server speed sits on the other side of the ledger. The guide states that if Google can load and render pages faster, it may read more content from the site, and recommends supporting HTTP 304 (Not Modified) so unchanged pages do not have to be downloaded again. The relationship between response time and crawling is described in the entry on TTFB.
How to protect it
The best practices from the crawl budget documentation, in order:
- Consolidate duplicate content so crawling focuses on unique content rather than unique URLs.
- Block URLs that should never be crawled with robots.txt: infinite scroll duplicates, sorted versions of the same page. Google warns that freed budget is not shifted elsewhere unless the capacity limit is already being hit.
- Return 404 or 410 for permanently removed pages. Google will not forget a URL, but a 404 is a strong signal not to crawl it again, while blocked URLs stay in the queue much longer.
- Eliminate soft 404s, which are listed in the Page indexing report; see the entry on soft 404.
- Keep sitemaps current and include
<lastmod>for updated content; the format is in the entry on the XML sitemap. - Avoid long redirect chains.
- Make pages fast and support 304 responses.
To get more budget, the guide offers two routes: add server resources if hostload is the limit, and improve the content's popularity, user value and uniqueness.
How to check it
The Crawl Stats report in Search Console, under Settings, shows total crawl requests, total download size, average response time, host status, and breakdowns by response code, file type, crawl purpose and Googlebot type. Google says sites with fewer than a thousand pages should not need it. Useful reads:
- host status: whether robots.txt fetching, DNS or server connectivity had significant failures in the last 90 days,
- crawl responses: the share of 200 versus 5xx, 404, redirects and "robots.txt not available",
- crawl purpose: discovery versus refresh, which shows whether new content is being found,
- average response time, which maps directly to the capacity limit.
Server logs give the raw version: count Googlebot requests per day, directory and status code, after verifying them with Google's verification procedure. The status codes Google expects for each situation are in the entry on HTTP status codes.
For a quick check, run curl -sIL on a sample of parameter URLs from the logs and see whether they return 200 with duplicate content, redirect in chains, or are blocked.
How IndexRise monitors it
IndexRise does not read Googlebot's activity, but it records every night the signals that move the crawl capacity limit: the status code of each monitored URL, the response time with a warning when it rises to three times the recent median, robots.txt availability, redirect chains and loops, and sitemap health including the URL count. A night of 5xx responses, a robots.txt that stops responding or a sitemap that drops half its URLs are the events that make Google crawl less, reported before the Crawl Stats report shows the dip.
Frequently asked questions
Does my site have a crawl budget problem?
Probably not unless it is large. Google's guide targets sites with over a million unique pages that change weekly, sites with more than 10,000 pages that change daily, and sites where a large share of URLs sit in Discovered - currently not indexed.
Does noindex save crawl budget?
No. Google's crawl budget documentation says it will still request the page and only drop it after seeing the noindex, which wastes the crawl. Use robots.txt to block URLs you never want crawled.
Does a 404 waste crawl budget?
Less than a block does. Google describes a 404 or 410 as a strong signal not to crawl the URL again, while URLs blocked in robots.txt stay in the crawl queue much longer.
Can I buy more crawl budget?
No. Google's documentation names two ways to get more: add server resources if hostload is the limit, and improve content quality, popularity and uniqueness, which raise crawl demand.
Does site speed affect crawl budget?
Yes. Google states that if the site responds consistently and response times stay stable or improve, the crawl capacity limit goes up, and that slower responses, 5xx errors or 429 signals make it go down.
Is crawl budget per domain or per host?
Per hostname. Google's documentation treats https://www.example.com/ and https://code.example.com/ as separate sites with separate crawl budgets.
Updated: 2026-10-20