Definition
Plain language
Wobble in a measurement that everyone treats as meaningless and averages away.
As stated in the literature
Variation in an outcome attributed to experimentally irrelevant factors and handled by averaging or marginalizing; the claim here is that prompt-induced nuisance variance is structured and therefore exploitable rather than random.
Also called: nuisance variation
Why it matters: Treating that wobble as meaningless noise is dangerous if it turns out to be structured, because then someone can deliberately steer it.
For example, a research team runs the same test with five different wordings, sees the scores bounce around, and just reports the average.
Heard on the show
“And the standard response is: that's nuisance variance.”Episode 243 — How a Hundred Meaningless Word Choices Add Up to Flip a Model's Answer