Definition
Plain language
A grading rule for probability forecasts that can't be gamed — your best expected score comes from reporting what you actually believe.
As stated in the literature
A scoring function for probabilistic predictions whose expected value is maximized by reporting one's true distribution; Brier, log score, and CRPS are examples, differing in their sensitivity to upper-tail miscalibration.
Also called: proper scoring rules
Why it matters: It rewards honest probability estimates and removes the incentive to game predictions, which is essential for trustworthy forecasting and calibration.
For example, a weather forecaster graded this way earns their best long-run score only by stating the true chance of rain rather than exaggerating to look confident.
Heard on the show
“They recommend that every LLM forecasting benchmark report at least one tail-integrating proper scoring rule — CRPS, log score, something — alongside whatever threshold metrics they're already using.”Episode 069 — When Smarter Models Forecast Worse: The Hidden Failure Mode in LLM Predictions