Definition
Plain language
A grading rule for forecasts that checks one yes-or-no question and asks whether you put about the right probability on it.
As stated in the literature
A proper scoring rule for probabilistic predictions of a binary outcome, equal to the squared error between predicted probability and realized indicator; standard in event-prediction benchmarks but blind to upper-tail miscalibration on distributional forecasts.
Why it matters: It's a simple, proper scoring rule for probability forecasts that rewards calibration, though it can miss problems in the tails of richer distributional predictions.
For example, if you forecast 70% chance of rain and it rains, your Brier score for that day is (1 − 0.7)² = 0.09.
Heard on the show
“The reward uses the Brier score, which is the classic tool for grading probabilistic forecasts — it goes back to weather forecasting in 1950.”Episode 183 — Why You Can't Fine-Tune Foresight Into an AI Agent