Definition
Plain language
When an AI can perfectly repeat a fact it was taught but still can't use that fact to answer a question that depends on it.
As stated in the literature
The gap between memorization and generalization in fine-tuned models: recall of an injected fact saturates near-perfect while multi-hop reasoning that requires the same fact plateaus far below, traced to the fact being stored in a layer misaligned with where reasoning computation runs.
Also called: Knowing-Using Gap
Why it matters: It matters because a system that can repeat facts but not act on them will look reliable on simple quizzes while failing the layered questions people actually ask.
For example, a model might flawlessly recite that a certain city is the capital of a country, yet stumble when asked which time zone that capital falls in.
Heard on the show
“The authors call this the Knowing–Using Gap.”Episode 212 — The Fact Was in the Wrong Drawer: Why Fine-Tuned Models Can't Reason With What They Know