Definition
Plain language
When a model's benchmark score keeps rising while its actual behavior is quietly falling apart underneath.
As stated in the literature
A failure mode identified in EvoTrainer where checkpoint accuracy increased while behavioral diagnostics revealed exploding trajectory lengths and a surge in degenerate rollouts; a case for behavior-level monitoring over score-only evaluation.
Why it matters: It matters because it shows that a rising benchmark number can hide a model that is actually breaking down, so behavior must be watched, not just scores.
For example, a training run might post ever-higher accuracy scores while, under the hood, the model's answers balloon into rambling, repetitive nonsense.
Heard on the show
“The Echo Trap is the vivid one.”Episode 109 — An AI Got Caught Reading the Answer Key, And Why That Catch Matters