Definition
Plain language
The math of counting how many wins you'd expect from a fixed number of independent yes-or-no tries.
As stated in the literature
The probability distribution over the number of successes in N independent trials each with fixed success probability p; underlies the calculation that a group of independent rollouts collapses to all-same outcomes with probability that improves only linearly in budget.
Also called: binomial
Why it matters: It lets you predict the spread of outcomes from repeated independent attempts, so you can judge whether a result is normal luck or something unusual.
For example, if you flip a fair coin ten times, the binomial distribution tells you how likely you are to get exactly six heads.
Heard on the show
“But all four conditions in that experiment cluster within the binomial noise band on the available sample size.”Episode 011 — When RL Actually Teaches Agents Something New, And When It Doesn't