Definition
Plain language
A number puzzle where you combine a few given numbers with arithmetic to hit a target, used to test reasoning models.
As stated in the literature
An arithmetic reasoning task in which a model must construct an expression from a set of numbers that evaluates to a target value; used as an RL training and evaluation benchmark with controllable difficulty.
Why it matters: Its difficulty can be dialed up or down precisely, making it a handy controlled test of a model's step-by-step reasoning.
For example, given the numbers 3, 7, and 50 and a target of 71, the model has to find that 3 times 7 plus 50 equals 71.
Heard on the show
“On the hard version of Countdown — that's the arithmetic puzzle from the opening example, hard meaning more numbers, larger target — vanilla GRPO gets about nineteen percent.”Episode 081 — When Reasoning Models Decide Before They Think: Detecting and Fixing Premature Confidence