Definition
Plain language
A moment when the results in front of you can't explain what actually happened, so you need a better way to look before deciding what to do next.
As stated in the literature
In EvoTrainer, the trigger for evolving the training harness itself — a point where existing metrics can't explain an outcome, can't distinguish competing hypotheses, or can't justify the next action — prompting the trainer to add new diagnostics rather than just tweak the recipe.
Why it matters: It flags when your current measurements aren't enough to make a good decision, prompting better instrumentation instead of blind guessing at fixes.
For example, a model's training run stalls but the usual charts look normal, so the trainer realizes it needs a new measurement before it can figure out what went wrong.
Heard on the show
“… And the trigger for the harness to evolve is what they call a diagnostic gap — a moment where the evidence you have can't explain the outcome, or can't tell two hypotheses …”Episode 109 — An AI Got Caught Reading the Answer Key, And Why That Catch Matters