Definition
Plain language
A benchmark of hard competitive-programming problems used to test AI coding ability.
As stated in the literature
A competition-level coding benchmark of algorithmic problems with hidden test cases; used to evaluate multi-agent systems like EvoChamber, where single agents solve under 7% and evolved pools reach roughly 35%.
Why it matters: It provides a demanding, verifiable test of whether an AI can actually solve hard programming problems rather than just producing plausible-looking code.
For example, a CodeContests problem might ask for a program that finds the shortest route through a maze, then check it against hidden test cases.
Heard on the show
“… Competition coding is starker: a single agent solves under seven percent of CodeContests problems, and the evolved pool solves thirty-five — five times over — while the memory-based baselines …”Episode 200 — The One Mechanism That Turns Twenty AI Clones Into an Actual Team