Definition
Plain language
The chance that a model gets a problem right at least once when given a fixed number of tries.
As stated in the literature
The probability that at least one of k independent samples from a model produces a correct answer, often estimated using an unbiased hypergeometric estimator.
Also called: pass-at-k, pass@k, pass at 1, pass at 3, pass at 8
Why it matters: It captures the practical reality that you often get multiple attempts, and exposes models that have the answer in them but rarely on the first try.
For example, pass@10 of 80 percent means in ten tries the model produces at least one correct solution 80 percent of the time.
Heard on the show
“… And if you want the full transcript with every term defined inline — the Lean stuff, the pass-at-k stuff — plus the links over to other episodes that touch these same ideas, that all lives on paperdive …”Episode 117 — How an Open AI System Verified 672 Hard Math Proofs for Under $300