Definition
Plain language
A bundle of standard tests used to score how good a language model is overall.
As stated in the literature
An aggregate suite of downstream language-modeling evaluations used alongside Lambada to measure general capability, e.g., in Attractor Model comparisons.
Why it matters: It gives a shared scorecard so different models can be compared on general ability rather than just one narrow skill.
For example, a team can run their new model through this bundle of tests to get a single overall sense of how capable it is.
Heard on the show
“… On downstream evaluations — Lambada and the CORE benchmark suite — the paper reports relative improvements up to nearly twenty percent over a parameter-matched …”Episode 041 — When the Iteration Teaches the Model to Skip the Iteration