Definition
Plain language
A wide-ranging multiple-choice benchmark covering many academic subjects.
As stated in the literature
Massive Multitask Language Understanding, a benchmark of multiple-choice questions across 57 subjects used to evaluate broad knowledge in language models.
Also called: M-M-L-U
Why it matters: It serves as a broad, widely-cited check on whether a model has general knowledge across many academic fields.
For example, MMLU might ask a model a question on constitutional law and another on organic chemistry mechanisms.
Heard on the show
“That's the standard computer-use benchmark, the GUI agent equivalent of MMLU.”Episode 080 — How a Two-Agent Trick Unlocked Large-Scale Training for Computer-Use Agents