Definition
Plain language
A statistics trick that re-samples your data many times over to check whether a measured difference is real or just luck.
As stated in the literature
A resampling method that estimates the variability of a statistic by repeatedly drawing with replacement from the observed sample; the paired variant keeps yoked scores together to test whether a per-item gap stays positive across resamples.
Also called: paired bootstrap, bootstrapping
Why it matters: It matters because it lets you judge whether a measured difference is trustworthy even when you can't collect more data.
For example, to check whether one model really beats another, you can reshuffle and re-sample the test results thousands of times and see how often the winner still comes out ahead.
Heard on the show
“The method is called the Agentic Bootstrap, after the classic statistical bootstrap: when you can't afford the real do-over, you simulate it.”Episode 196 — AI Agents Reached Opposite Conclusions From the Same Data — and Passed Review