Definition
Plain language
The amount a measurement naturally wobbles by chance, below which you can't tell whether a change was real.
As stated in the literature
The variance in a metric due to stochastic factors like random seeds and data ordering; improvements smaller than the noise floor cannot be reliably distinguished from luck.
Why it matters: It sets the bar for believing a result, since improvements smaller than the natural wobble could be pure luck.
For example, if a model's score wobbles by a point just from changing the random seed, a one-point gain doesn't prove anything real happened.
Heard on the show
“The authors are reasonably honest about it — they acknowledge the dependence — but they don't measure the noise floor.”Episode 019 — When the Best Reward Model Trains the Worst Policy: Inside EvoLM