Robots.txt
A simple text file at the root of a website that tells crawlers which pages they may or may not access — a voluntary set of rules, not a technical lock.
Full Definition
The robots.txt file is a standard (the Robots Exclusion Protocol) that allows website operators to communicate crawling permissions to automated agents. Located at the root URL (e.g. https://example.com/robots.txt), it specifies which user agents are allowed or disallowed from crawling which paths, and can set a Crawl-Delay. Search engine bots like Googlebot honour robots.txt scrupulously. Scrapers are not technically prevented from ignoring robots.txt — it is advisory, not enforced. However, ignoring it may violate a site's terms of service and, in some jurisdictions, has legal implications. Responsible scrapers read and respect robots.txt.
