Definition
Plain language
A small file on a website that tells automated programs which pages they're allowed to visit.
As stated in the literature
A site-root text file specifying crawl permissions for automated agents under the Robots Exclusion Protocol; advisory rather than enforced, and one of the first things a scraping or browsing agent reads — or ignores.
Also called: robots file
Why it matters: It is the standard, if unenforced, signal of where automated agents may go, so whether a scraper honors it shapes the ethics of web crawling.
For example, a news site can list its private archive in this file to ask crawlers to stay away from those pages.
Heard on the show
“It scrapes the site's robots file and sitemap.”Episode 061 — When Helpful Agents Go Sideways: A 404 Error, Campus Security, and Why Alignment Misses This