HTTP Headers
Small pieces of metadata sent with every web request and response, describing things like the browser, content type, and cookies — quietly shaping how servers treat your traffic.
Full Definition
HTTP headers are key-value pairs attached to every HTTP request and response that carry metadata about the message. Request headers include User-Agent (identifies the browser/client), Accept-Language, Cookie, Referer, Origin, and Accept-Encoding. Response headers include Set-Cookie, Content-Type, Cache-Control, and security headers like X-Frame-Options. For scrapers, crafting realistic request headers is critical: missing or incorrect headers are a primary bot-detection signal. Headers like X-Forwarded-For and Via must be stripped by elite proxies to avoid revealing that a proxy is in use.
