pass at k

Definition

The chance that a model gets a problem right at least once when given a fixed number of tries.

The probability that at least one of k independent samples from a model produces a correct answer, often estimated using an unbiased hypergeometric estimator.

Also called: pass-at-k, pass@k, pass at 1, pass at 3, pass at 8, pass-at-k-T

Mentioned in 2 episodes

1. 026
   What RL Actually Does to Language Models, at the Token Level
2. 011
   When RL Actually Teaches Agents Something New, And When It Doesn't

Related concepts