Definition
Plain language
When the situations a system meets in real use drift away from the examples it learned on, so it starts making mistakes.
As stated in the literature
A mismatch between the training distribution and the deployment or visited-state distribution; the core reason behavior cloning compounds error, since flawless experts never demonstrate recovery from off-distribution states.
Why it matters: It matters because systems that look flawless in training can fail badly in the wild once real conditions drift from what they saw.
For example, a self-driving model trained only on smooth highway footage may flounder the first time it hits a snowy back road.
Heard on the show
“5 percent may be more brittle on distribution shift than it looks.”Episode 047 — When Agent Benchmarks Lie: The Harness Problem in Open-Source AI