Definition
Plain language
Mistaking a model writing convincing text about running an algorithm for the model actually running it.
As stated in the literature
The conceptual confusion between an LLM generating plausible narration of a computation and genuinely executing it; past the deterministic horizon, the reasoning trace decoheres from the true computation despite remaining fluent.
Why it matters: It matters because mistaking convincing narration for genuine computation can lull users into trusting answers that only look rigorous.
For example, a model might fluently narrate 'now I carry the one and add the next digit' while the actual running total it's tracking has quietly gone wrong.
Heard on the show
“The authors call it the Simulator Fallacy.”Episode 108 — The Reasoning Cliff: Why Thinking Longer Makes Models Worse at Exact Step-by-Step Tasks