Brier scores measure forecast accuracy and calibration
Priya Nandakumar, Writer (AI-assisted) ยท
The Brier score lets you measure how accurate your probability forecasts really are, not just whether you called it right or wrong. Unlike simple accuracy, it rewards you for being honest about your confidence and penalizes you for over- or under-stating your chances. Here’s how to use it to grade your own predictions—and what that tells you about your calibration.
The Brier score: what it is and when to use it
The Brier score calculates the mean squared difference between your predicted probability (from 0 to 1) and the actual outcome (0 if it doesn’t happen, 1 if it does)[1]. In other words, it tracks how far off your confidence was from reality on average. A perfect Brier score is 0 (all predictions spot on), while 1 is the worst possible for a binary event.
When to use it:
- You give a probability (like 70%) that an event will happen.
- The event is binary or fits into discrete categories (like “rain” or “no rain”)[1].
- You want to reward good calibration, not just lucky guesses.
It is not for ordinal outcomes (like rankings) or when you only give yes/no predictions without probabilities[1].
Why Brier scores are fair: strictly proper scoring
A scoring rule is strictly proper if it incentivizes you to report your true beliefs: you minimize your expected Brier score only by being honest about your odds[2]. This means you’re not tempted to fudge your numbers to look better after the fact. If your forecasted probabilities match reality over time, your Brier score will be as low as possible[1,2].
This property makes the Brier score useful for:
- Comparing different forecasters or models on a level playing field.
- Training models that need to output calibrated probabilities.
- Auditing your own judgment for overconfidence or underconfidence.
How to calculate it: step by step
For each forecasted event, you need:
- The predicted probability you assigned (e.g. 0.7 for 70%).
- The actual outcome (1 if it happened, 0 if not).
The Brier score for one forecast is (predicted probability – actual outcome) squared[1]. For a set of predictions, add up each squared difference and divide by the total number of forecasts:
Brier score = (1/N) × sum of (predicted probability – actual outcome)²
So a forecast of 1 (100% certain) that is correct scores 0; if wrong, it scores 1 (worst). A 50% forecast always scores 0.25, whether right or wrong[1].
A worked example
Suppose you forecast the chance of rain for five days, like this:
| Day | Your forecast (%) | Predicted probability | Did it rain? | Actual outcome | (Forecast - Outcome)² |
|---|---|---|---|---|---|
| 1 | 70 | 0.7 | Yes | 1 | (0.7 - 1)² = 0.09 |
| 2 | 30 | 0.3 | No | 0 | (0.3 - 0)² = 0.09 |
| 3 | 90 | 0.9 | Yes | 1 | (0.9 - 1)² = 0.01 |
| 4 | 40 | 0.4 | Yes | 1 | (0.4 - 1)² = 0.36 |
| 5 | 60 | 0.6 | No | 0 | (0.6 - 0)² = 0.36 |
Sum of squared differences: 0.09 + 0.09 + 0.01 + 0.36 + 0.36 = 0.91
Brier score = 0.91 / 5 = 0.182
What this tells you: The closer your Brier score is to zero, the more accurate and better calibrated your probability forecasts are. In this set, your high-confidence misses (day 4 and 5) hurt your score most—each squared error for a confident but wrong forecast is a visible dent in your average.
What Brier scores reveal about your forecasting
- Calibration: The Brier score’s structure means any gap between your prediction and what actually happened increases your penalty by the square of that gap. This design directly punishes overconfidence: the further your forecasted probability is from the outcome (0 or 1), the steeper the cost.
- Comparing models or people: Averaging Brier scores across many predictions allows direct comparison of calibration and sharpness between forecasters or models[2]. A lower average Brier score signals better-aligned probabilities with reality.
- Honest probabilities: Because the Brier score is a strictly proper scoring rule, it rewards you only when you state your true beliefs[1][2]. There’s no advantage in hedging or “playing it safe”; the expected score is minimized only by giving your actual estimate.
No source gives a benchmark figure for “good” or “random” Brier scores; what the sources do show is that tracking your Brier scores over time is the practical way to see if your probability judgments actually match outcomes. If your average Brier score consistently drops as you refine your process, that’s evidence your forecasts are getting better.
Method and Sources: [1] Brier score - Wikipedia, [2] Scoring rule - Wikipedia
Questions readers ask
How do I interpret my Brier score?
A Brier score of 0 is perfect; 1 is the worst for a binary event[1]. If you always guess 50%, you’ll get 0.25 on average. Scores below 0.25 mean you’re adding value; above 0.25 suggest you’re miscalibrated or overconfident.
Can I use the Brier score for events with more than two outcomes?
Yes, but you must use the original Brier formulation for multi-class outcomes, which sums squared differences across all possible categories[1]. Each event must still have probabilities assigned that sum to one.
What if I only make yes/no forecasts, not probabilities?
The Brier score is not designed for yes/no predictions—it measures the quality of your probability estimates[1]. For yes/no, use accuracy or other classification metrics instead.