Definition
Plain language
A medical question benchmark used to test how well models reason about health topics.
As stated in the literature
A medical-reasoning evaluation suite used in metacognition-as-reward experiments to test whether process-supervised reasoning gains transfer to expert health domains.
Why it matters: It tests whether a model's reasoning improvements actually carry over into a demanding expert field like medicine, where mistakes have real stakes.
For example, it might ask a model to reason through a patient's symptoms and pick the most likely diagnosis.
Heard on the show
“5-9B model gets slightly worse on Frontier Science, on Research QA, on a medical evaluation called LLMEval-Med.”Episode 079 — An Old Idea From Cognitive Psychology Reshapes How We Reward Reasoning Models