Definition
Plain language
A small automated check that a single piece of code — or, here, a single definition — behaves the way it's supposed to.
As stated in the literature
A test asserting a property that should hold if a component is correct; in autoformalization, auxiliary lemmas serve as unit tests for a new formal definition, since a definition's meaning can't be checked by the compiler directly.
Also called: unit tests
Why it matters: It matters because these small checks catch broken behavior early, and can even verify a formal definition means what its author intended.
For example, a test might feed a sorting function an unsorted list and check that it comes back in order.
Heard on the show
“Models now write the unit tests, the eval rubrics, the answer keys, and increasingly the machine-checkable criterion sitting inside a reinforcement learning loop.”Episode 233 — Why a Model Can Grade an Answer But Not Write the Answer Key