Definition
Plain language
A number showing how much a measured average might bounce around if you repeated the measurement.
As stated in the literature
The estimated standard deviation of a sample statistic's sampling distribution; used to judge whether an observed performance difference is distinguishable from noise.
Also called: standard errors
Why it matters: It tells you how much to trust a single number, and keeps small gaps between systems from being read as real differences.
For example, a reported score of 88% with a standard error of 2 points means a repeat test would plausibly land anywhere from the mid-80s to the low 90s.
Heard on the show
“On 1,069 unused GSM8K problems, which are grade-school math word problems, it scored about six percentage points BELOW the base model, with a reported standard error of about two points.”Episode 291 — Certifying a Neural Network's Arithmetic by Taking Away Its Memory