Definition
Plain language
A science benchmark built from twin problems that share the same solution steps but need completely different facts, to test whether AI improved its reasoning or just its recall.
As stated in the literature
A benchmark of 144 isomorphic problem pairs across physics, chemistry, biology, and earth science, each pair sharing procedural structure but disjoint knowledge; used to separate reasoning gains from knowledge gains by checking whether a reasoning-mode improvement transfers to the structurally identical twin.
Why it matters: It helps tell apart whether an AI got better at genuine reasoning or merely memorized more facts, which are easy to confuse from a single test score.
For example, IsoSci might pair a physics problem and a chemistry problem that need the exact same solution steps but completely different facts, then check whether a model that improved on one also improves on its twin.
Heard on the show
“On a new benchmark called IsoSci, covering the same sciences, it loses by about twenty-five.”Episode 197 — Twin Problems Suggest AI Reasoning Gains Are Mostly Better Fact Recall