Definition
Plain language
A probability distribution describing how many of a certain kind you'd draw if you sampled without replacement.
As stated in the literature
A discrete probability distribution over the number of successes in a sample drawn without replacement from a finite population, used in the unbiased pass-at-k estimator.
Why it matters: It underlies the standard unbiased estimator for pass@k, so getting the math right matters whenever you score a model on multiple samples per problem.
For example, if a bag has 10 red and 90 blue marbles and you draw 20 without replacing them, the hypergeometric distribution tells you the chance of getting exactly 3 reds.
Heard on the show
“The math is hypergeometric — we don't need it.”Episode 011 — When RL Actually Teaches Agents Something New, And When It Doesn't