A model can produce confident-looking probabilities and still be wrong in a systematic way. Calibration is the process of checking whether the probabilities a model gives actually match the frequency of real outcomes over time.
What calibration means
A well-calibrated model is one where outcomes happen at roughly the frequency the model predicted. If a model assigns a 70% win probability to a large number of matches, around 70% of those players should win. If only 60% do, the model is overconfident. If 80% do, it is underconfident.
Calibration is different from accuracy. A model can pick the winner more often than not and still be badly calibrated, if its probabilities are too extreme or too timid.
How to test calibration
The standard approach is to group predictions into probability bands and compare the average prediction in each band with the actual win rate.
| Predicted Probability Band | Matches | Average Prediction | Actual Win Rate | Verdict |
| 50–60% | 800 | 55% | 54% | Well calibrated |
| 60–70% | 650 | 65% | 63% | Slightly overconfident |
| 70–80% | 420 | 75% | 71% | Overconfident |
| 80–90% | 210 | 85% | 78% | Noticeably overconfident |
| 90%+ | 90 | 93% | 84% | Strongly overconfident |
(Illustrative figures.)
In this example the model is reasonably reliable in the middle of the range but becomes progressively overconfident at the high end, a common pattern when models extrapolate from limited data.
Scoring rules
Beyond banded tables, analysts use scoring rules to summarise performance in one number:
- Brier score: the average squared difference between predicted probability and the actual outcome (1 or 0). Lower is better.
- Log loss: penalises confident wrong predictions more heavily. Also lower is better.
Both can be compared against a simple baseline, such as using market-implied probabilities, to judge whether the model adds information.
Why sample size matters
Calibration checks need many observations. A band containing only 20 matches can deviate widely from its expected rate by chance alone. As a rough guide, hundreds of observations per band give a more reliable picture, and thousands give a clearer view still.
Out-of-sample testing
A model tested only on the data it was built from will almost always look better than it really is. This is called overfitting: the model has learned the quirks of past data rather than the underlying patterns. A more honest test uses out-of-sample data, such as results from a period the model never saw during development.
A simple discipline is to split data by time: build the model on earlier seasons and test on later ones. This mimics real use, where the model has to predict future matches.
Calibration by segment
Overall calibration can hide problems in specific segments. A model might be well calibrated on hard courts but overconfident on clay, or accurate for ATP main-draw matches and unreliable for Challenger events. Checking calibration by surface, tour, tournament tier and rating gap can reveal where the model needs adjustment.
Comparing against the market
A useful benchmark is whether a model’s probabilities beat the closing market price in terms of scoring rules. Markets incorporate a great deal of information, so outperforming them consistently is a high bar. A model that is well calibrated but no better than the market can still be useful for understanding matches, but it offers limited scope for identifying value.
Practical adjustments
If calibration reveals systematic bias, several fixes are common:
- Shrinking extreme probabilities toward 50%.
- Fitting a simple recalibration curve to map raw outputs onto observed frequencies.
- Adding more data for the weak segments.
- Down-weighting small-sample inputs.
A common failure: overconfidence at the extremes
Many models are well behaved in the middle of the probability range and drift at the edges. Very high probabilities are rare events by construction, so there are few examples to learn from, and small errors in the inputs can turn a 90% estimate into a real 80%. This matters because extreme probabilities are exactly where large stakes or very short prices tend to appear.
Calibration is not a one-off task
Calibration can degrade over time. Playing styles evolve, tour conditions change, and data sources are revised. A model that was well calibrated three seasons ago may have drifted without anyone noticing. Re-running calibration checks on a regular schedule, for example each season or after each major change in the data pipeline, catches this early.
What good looks like
No model will be perfectly calibrated. A realistic goal is to be close to the diagonal in each probability band, with deviations that are small relative to the sampling noise. If deviations are consistent in one direction across several bands, that is a signal worth investigating, rather than a reason to ignore the result.
Summary
Calibration asks a simple question: do the probabilities mean what they say? Testing predictions in banded groups, using scoring rules, and checking out-of-sample and by segment gives a far more honest picture of model quality than counting winners alone.
