Definition
Plain language
A grading rule for forecasts that checks the whole shape of a predicted range, not just one cutoff.
As stated in the literature
Continuous Ranked Probability Score, a proper scoring rule that integrates squared error between a forecast CDF and the realized outcome across all thresholds; sensitive to tail miscalibration where Brier-style metrics are not.
Also called: Continuous Ranked Probability Score
Why it matters: It rewards forecasters for getting the whole shape of uncertainty right, which simple yes/no scoring rules miss.
For example, a forecaster predicting tomorrow's temperature as a full distribution is scored by integrating squared error between that distribution and the actual temperature across every possible threshold.
Heard on the show
“The second grading philosophy is called CRPS — the Continuous Ranked Probability Score.”Episode 069 — When Smarter Models Forecast Worse: The Hidden Failure Mode in LLM Predictions