Definition
Plain language
A benchmark of realistic coding tasks used to test code-writing models.
As stated in the literature
A coding benchmark targeting practical Python programming tasks with library usage and integration challenges.
Also called: big-CODE-bench
Why it matters: Real-world coding is mostly about gluing libraries together correctly, and benchmarks like BigCodeBench measure that better than toy algorithm problems.
For example, BigCodeBench tasks might require correctly chaining together calls to pandas, requests, and json to produce a working script.
Heard on the show
“The other is AI research and development tasks, including kernel optimization on KER-nel-bench and code generation on big-CODE-bench.”Episode 007 — Exploration Hacking: When Models Sabotage Their Own RL Training