How We Grade and Report Prediction Accuracy

Any service can claim accuracy. Very few publish the method by which they measure it, which is what makes the claim checkable. This page is that method.

What gets graded

Every prediction published on MatchSense is recorded before kick-off with its market, selection, probability, the associated number available at publication, and a timestamp. After the match, it is graded against the final result.

Nothing is excluded. Losing predictions are graded identically to winning ones, and no competition, market or period is dropped from the sample after the fact. The reason is simple: the moment a record becomes selective, every metric computed from it becomes meaningless, and that is the single most common way accuracy claims are inflated.

The metrics we track

Brier score. The mean squared difference between the predicted probability and the outcome, where the outcome is 1 if the event happened and 0 if it did not. Lower is better; a perfect forecaster scores 0. Its important property is that it punishes confident errors far more than uncertain ones — predicting 95% on something that does not happen costs much more than predicting 55%.

Log loss. A stricter alternative that penalises confident errors even more aggressively. It is useful precisely because it is unforgiving of the failure mode that matters most: overconfidence.

Calibration by band. Predictions are grouped into confidence bands and the realised hit rate in each band is reported alongside the predicted rate:

Reporting by band matters because an aggregate average can hide systematic overconfidence at the high end, which is exactly where the model's most confident predictions are made and exactly where a miscalibrated model does the most damage.

What we deliberately do not publish

Win rate without the accompanying probability. A hit rate quoted alone is not an accuracy measure. It can be trivially inflated by predicting heavy favourites, and it says nothing about whether following the predictions is profitable.

Streaks. "Nine winners in a row" describes variance. Every strategy with a positive win rate produces streaks, and so does every strategy with a negative one.

ROI at unavailable figures. Return calculated at the best figure found anywhere after the fact, rather than at the figure available when the prediction was published, is a fiction. Ours is computed at recorded publication figures.

Aggregate accuracy across markets. Pooling a 1X2 hit rate with a total goals hit rate produces a number that means nothing, because the base rates differ. Metrics are reported per market.

What we would expect you to demand

The checklist in how to evaluate a football prediction service was written to be applied to us as much as to anyone else. A complete timestamped record, a sample of several hundred predictions, ROI at the recorded figures, and calibration by band. If any of those is missing from a service you are paying for, ask why.

Why this is published at all

There is a commercial argument against publishing a method that makes your own numbers falsifiable. We take the opposite view: a prediction service whose accuracy cannot be checked is indistinguishable from one that is guessing, and the only durable way to be distinguishable is to be checkable.

The modelling pipeline that produces the predictions being graded here is documented in how the MatchSense prediction model works.

Related reading

• Prediction performance — the graded record
• How the MatchSense model works
• The models behind the predictions