Definition
Plain language
A number puzzle where you combine a few given numbers with arithmetic to hit a target, used to test reasoning models.
As stated in the literature
An arithmetic reasoning task in which a model must construct an expression from a set of numbers that evaluates to a target value; used as an RL training and evaluation benchmark with controllable difficulty.
Why it matters: Its difficulty can be dialed up or down precisely, making it a handy controlled test of a model's step-by-step reasoning.
For example, given the numbers 3, 7, and 50 and a target of 71, the model has to find that 3 times 7 plus 50 equals 71.
Heard on the show
“… On a math puzzle benchmark called Countdown, and on reducing sycophancy — telling users what they want to hear instead of the truth — the best …”Episode 199 — Finding a Model's Hidden Behaviors Without Knowing What You're Looking For