Crawl Budget
The number of pages a search engine or crawler will fetch from a site within a given time, limiting how much of a large website gets crawled and indexed.
Full Definition
Crawl budget is the number of URLs a crawler (whether Googlebot or your own spider) will fetch from a domain in a given window. For search engines, crawl budget is influenced by the site's crawl rate limit (how fast the server can handle requests) and crawl demand (how popular and frequently updated the content is). For custom scrapers, managing crawl budget means prioritising high-value pages, avoiding duplicate content paths, and respecting the server's capacity to prevent triggering rate limits or IP blocks. Efficient crawlers use sitemaps, canonical tags, and robots.txt to focus their budget.
