Definition
Knowledge vs. reasoning is a distinction between a model's ability to recall memorized facts and its ability to execute the procedural, step-by-step logic needed to solve a problem. One methodology for isolating the two constructs matched pairs of “isomorphic” problems that share identical logical structure but differ in surface content, so that a gap in performance between the original and its twin can be attributed to reliance on memorized knowledge rather than transferable reasoning.
Episodes covering this
Worth reading next
Papers we haven't done a deep dive on yet, but would recommend on this topic.