Definition
Plain language
How much a model's answer changes when you reword the question without changing what it means.
As stated in the literature
Variation in model outputs under meaning-preserving perturbations of the input (ordering, formatting, synonyms, typos); typically reported as variance across prompt templates and treated as an evaluation reliability issue.
Also called: prompt-sensitivity
Why it matters: It means a single test result may say more about how the question was worded than about what the model actually knows.
For example, asking "which is safer?" versus "which is riskier?" about the same two options gets you inconsistent answers even though the underlying question is identical.
Heard on the show
“And if it does, then prompt sensitivity stops being noise you average over, and becomes a control surface somebody can grab with a few thousand queries.”Episode 243 — How a Hundred Meaningless Word Choices Add Up to Flip a Model's Answer