Definition
Plain language
A wide-ranging multiple-choice benchmark covering many academic subjects.
As stated in the literature
Massive Multitask Language Understanding, a benchmark of multiple-choice questions across 57 subjects used to evaluate broad knowledge in language models.
Also called: M-M-L-U
Why it matters: It serves as a broad, widely-cited check on whether a model has general knowledge across many academic fields.
For example, MMLU might ask a model a question on constitutional law and another on organic chemistry mechanisms.
Heard on the show
“On MMLU and ARC-Challenge, all three models — baseline, practical FPO, relaxed FPO — are within statistical noise.”Episode 025 — The Missing Gradient Term That Predicts Sycophancy in RLHF