Definition
Plain language
Whether a forecaster's higher scores really do land on the things that end up happening.
As stated in the literature
The ranking component of forecast skill, measured here by AUROC; separable from calibration, so a forecast can be well-calibrated in aggregate yet have below-chance discrimination.
Why it matters: A forecast can look honest in aggregate yet be useless — or actively misleading — for any individual decision if its higher numbers do not actually land on the events that occur.
For example, a weather forecaster who says 70% rain on the days it rains and 30% on the days it doesn't has good discrimination, while one who says 50% every day has none.
Heard on the show
“That's a discrimination problem the current training pipelines aren't solving.”Episode 061 — When Helpful Agents Go Sideways: A 404 Error, Campus Security, and Why Alignment Misses This