Definition
Plain language
A benchmark of yes-or-no questions that require chaining together several common-sense facts.
As stated in the literature
A multi-hop commonsense reasoning benchmark requiring implicit decomposition of questions into intermediate reasoning steps.
Why it matters: It tests whether a model can break an everyday question into hidden reasoning steps rather than just matching surface words.
For example, answering 'could a goldfish survive in the Dead Sea?' requires quietly combining facts about salinity and fish before saying yes or no.
Heard on the show
“On a benchmark called StrategyQA, you can ask a small Llama model a question and it gets the right answer about eighty-one percent of the time.”Episode 077 — Reading a Model's Confidence Curve to Decide When Chain-of-Thought Is Worth It