Definition
Plain language
How often something actually happens in the real world, before you start measuring anything else.
As stated in the literature
The unconditional frequency of an event in the population of interest; benchmarks that oversample rare events relative to their base rate systematically misestimate real-world impact.
Also called: base rates
Why it matters: Ignoring how often something really happens leads teams to celebrate or panic over behaviors that barely occur in deployment.
For example, if only one in a thousand customer emails is actually a refund request, a test set where half the emails are refund requests will make a system look far better or worse at handling them than it will be in practice.
Heard on the show
“The behavior may be real, but its *base rate* in actual production may be much lower than these headline numbers suggest.”Episode 001 — When AI Models Quietly Protect Each Other From Shutdown