Definition
Plain language
A small set of programming problems used as a standard test for code-generating models.
As stated in the literature
OpenAI's hand-written benchmark of Python programming problems with unit tests, used to measure functional code generation accuracy.
Why it matters: It became one of the default yardsticks for comparing code-generation models, shaping which systems get called 'good at programming.'
For example, a model might be asked to write a Python function that returns the longest common prefix of a list of strings, then graded on whether its code passes the hidden unit tests.
Heard on the show
“HumanEval problem one: split a string of nested parenthesis groups.”Episode 233 — Why a Model Can Grade an Answer But Not Write the Answer Key