Definition
Plain language
How sure a system says it is about its own answer.
As stated in the literature
A model-verbalized probability or certainty score, distinct from any internal likelihood; the standard object of calibration audits since Kadavath et al.
Also called: stated confidence, stated probability, stated probabilities
Why it matters: Users lean heavily on these stated numbers when deciding whether to double-check an answer, so a system whose confidence talk does not track its actual accuracy is quietly misleading them.
For example, a chatbot ends an answer with "I'm about 85% sure of this," which is its own stated estimate rather than anything measured inside the system.
Heard on the show
“And the consistency score is dead simple — how well does the model's stated probability fit its own flips?”Episode 152 — Training a Model to Mean What It Says, And Why That Isn't the Same as Being Good