Definition
Plain language
A standard benchmark of Python programming problems.
As stated in the literature
Mostly Basic Python Problems, a crowd-sourced benchmark of Python coding tasks with reference solutions and unit tests.
Why it matters: Alongside HumanEval, it's a default sanity check for code generation, especially for simpler, single-function tasks.
For example, an MBPP task might ask the model to 'write a function to find the n-th Fibonacci number,' which is then checked against a few unit tests.
Heard on the show
“GSM8K, MATH, HumanEval, MBPP, MultiArith — these are math and code benchmarks where the obvious topology is well-known and almost certainly present in the LLM's pretraining.”Episode 013 — Why Search Keeps Rediscovering the Same Workflow, and What That Means