Definition
Plain language
A statistics trick that re-samples your data many times over to check whether a measured difference is real or just luck.
As stated in the literature
A resampling method that estimates the variability of a statistic by repeatedly drawing with replacement from the observed sample; the paired variant keeps yoked scores together to test whether a per-item gap stays positive across resamples.
Also called: paired bootstrap, bootstrapping
Why it matters: It matters because it lets you judge whether a measured difference is trustworthy even when you can't collect more data.
For example, to check whether one model really beats another, you can reshuffle and re-sample the test results thousands of times and see how often the winner still comes out ahead.
Heard on the show
“But the loop currently requires a frontier teacher to bootstrap the subgoals and the progress labels.”Episode 008 — Why Long-Horizon AI Agents Get Stuck, and a Milestone-Based Fix That Helps