Definition
Plain language
The chance that a model gets a problem right at least once when given a fixed number of tries.
As stated in the literature
The probability that at least one of k independent samples from a model produces a correct answer, often estimated using an unbiased hypergeometric estimator.
Also called: pass-at-k, pass@k, pass at 1, pass at 3, pass at 8
Why it matters: It captures the practical reality that you often get multiple attempts, and exposes models that have the answer in them but rarely on the first try.
For example, pass@10 of 80 percent means in ten tries the model produces at least one correct solution 80 percent of the time.
Heard on the show
“And that's where the standard pass-at-k metric breaks down.”Episode 011 — When RL Actually Teaches Agents Something New, And When It Doesn't