Definition
Plain language
A Brier-style score computed from a full distributional forecast by picking a threshold.
As stated in the literature
A binary threshold metric reconstructed from quantile forecasts at a chosen cutoff, used in the inverse-scaling paper to demonstrate that the same outputs produce opposite capability-accuracy rankings under CRPS versus Brier-style scoring.
Why it matters: It illustrates that 'which model is better at forecasting' can hinge entirely on the scoring rule, with serious implications for evaluation claims.
For example, the same model's forecast distribution can rank highest under CRPS but lowest under derived Brier at a threshold of 0.5, just from how the metric collapses the distribution.
Heard on the show
“Under derived Brier from the same outputs, more capable models are better forecasters, rank correlation of plus point five.”Episode 069 — When Smarter Models Forecast Worse: The Hidden Failure Mode in LLM Predictions