Definition
Plain language
A standard set of competition math problems used to test reasoning models.
As stated in the literature
A 500-problem subset of the MATH benchmark, drawn from competition mathematics across algebra, geometry, and number theory.
Also called: MATH
Why it matters: It's one of the standard benchmarks reasoning-focused models report on, so it shapes which capabilities researchers actively chase.
For example, a MATH-500 problem might ask for the smallest positive integer satisfying a particular modular equation, requiring multi-step algebraic reasoning.
Heard on the show
“The headline number is just over seventy-nine percent on MATH-500 — a standard competition-math benchmark — versus about fifty-four percent for the best comparable prior latent method.”Episode 141 — How Two Tokens Reopened a Reasoning Method the Field Had Given Up On