Term
Crawl budget
The number of a site's pages a search engine is willing and able to crawl in a given time. On large sites it decides how fast new pages get indexed, and duplicates, broken links and endless filters waste it.
Updated
What it means
Crawl budget is the number of a site's pages a search engine is willing and able to crawl in a given time. It combines how many requests the server can handle without slowing down and how much the search engine wants the content, based on its popularity and freshness.
For a small site with a few hundred pages crawl budget hardly matters, as the crawler gets through them easily. It matters for large shops, catalogues and portals with tens of thousands of URLs, where it decides how quickly new products or articles reach the index.
The budget is also spent on URLs with no value: duplicate content, endless filter combinations, URL parameters, redirect chains and broken links. The crawler then wastes time on clutter and visits important pages less often.
How it works in practice
The first step is finding out where the crawler actually goes. The report in Search Console shows the number of requests and server response times, while server logs give a more precise picture of which URLs the crawler visits often and which it never touches.
Savings come from tidying up: filters with no search value are blocked in robots.txt, page variants are consolidated with a canonical URL and redirects are cut to a single hop. A fast server and lightweight pages let the crawler get through more.
An up-to-date XML sitemap with last-modified dates helps the crawler find new and changed pages. Internal linking matters just as much: a page that no link points to reaches the crawler slowly even if it is listed in the sitemap.
What to watch out for
Noindex and canonical tags do not save crawl budget directly, because the crawler must fetch the page to see the instruction. Robots.txt saves crawling, but a blocked page cannot pass on a canonical signal, so the tools need to be combined with care.
A sudden drop in crawled pages often signals a technical problem such as a slow server, 5xx errors or a firewall block. Fix it quickly, because after repeated errors the search engine slows its crawl rate and only restores it gradually.
Need the term explained on your own situation? Consulting from €55 / hour excl. VAT, depending on topic and complexity.