Definition
Plain language
A measure of how varied a model's answers are in meaning, not just wording.
As stated in the literature
A hallucination-detection metric that clusters multiple sampled responses by semantic equivalence and measures entropy over the clusters.
Why it matters: Wording diversity isn't the same as meaning diversity, and measuring entropy over meanings turns out to be a much sharper hallucination signal than measuring it over tokens.
For example, if a model gives ten answers that all paraphrase "Paris," entropy is low; if its answers split between "Paris," "London," and "Berlin," entropy is high — likely indicating hallucination.
Heard on the show
“And here's the part that should bother you: every hallucination detector we have — semantic entropy, internal probes, confidence scoring, all of it — looks at that answer and shrugs.”Episode 037 — Why Hallucination Detectors Miss Stale Facts: A Geometric Story About What Models Know But Don't Say