Crawl Rate
How fast a crawler sends requests to a website, usually measured in requests per second. Keeping it reasonable avoids overloading servers and triggering blocks.
Full Definition
Crawl rate is the frequency at which a web crawler or scraper sends HTTP requests to a target server, expressed as requests per second (RPS) or requests per minute (RPM). Crawling too fast can overload a server's resources (a form of accidental DoS) and will almost certainly trigger rate limiting or IP bans. Crawling too slowly wastes time and may miss time-sensitive data. The right crawl rate depends on the target site's size, server capacity, and bot-detection sensitivity. Best practice is to introduce random delays between requests (request throttling), distribute load across many IPs, and honour any Crawl-Delay directive in the site's robots.txt.
