How to measure whether a prediction model is any good
Hit rate cannot distinguish a good model from a cautious one. Proper scoring rules can. Brier score, log loss, calibration curves and the reliability–resolution decomposition, with the benchmarks that make each number interpretable.
Measure a prediction model with a proper scoring rule — Brier score or log loss — computed on out-of-sample events, and compare it against the de-vigged market price on the same events. Then check calibration: bucket every forecast by stated probability and verify that the 70% bucket happened about 70% of the time. Hit rate answers none of these questions.
Why can hit rate not measure a model?
A model that says "the favourite wins" on every match will post a high hit rate and contain no information beyond the league table. A model that says 52% on genuinely close games will post a mediocre hit rate and may be extremely valuable. Hit rate rewards confidence about easy questions.
Worse, hit rate is improper: a forecaster can raise it by rounding beliefs towards 0 and 1. Any metric that rewards exaggeration is unusable for a model whose output is a probability.
Brier score
Brier = (1/N) × Σ (p_i − o_i)²p = stated probability, o = outcome (1 or 0), N = number of forecasts.It is the mean squared error of a probability forecast. Lower is better; 0 is perfect.
| Brier score | What it means |
|---|---|
| 0.00 | Perfect — every forecast 100% correct and certain |
| 0.25 | What predicting 50% every time gives you. The floor of usefulness |
| above 0.25 | Worse than a coin flip stated honestly |
| 0.18–0.22 | Typical range for competitive football match forecasts |
For multi-outcome markets such as 1X2, the multi-class form sums the squared error across all outcomes, which makes the raw number not directly comparable to a binary Brier. Always state which form is being reported.
Log loss
Log loss = −(1/N) × Σ [ o_i × ln(p_i) + (1 − o_i) × ln(1 − p_i) ]The key property is asymmetric punishment: as a confident forecast approaches certainty and is wrong, the penalty approaches infinity. Saying 99% and being wrong is catastrophic; saying 60% and being wrong is a small cost.
For a bettor this is exactly the right incentive, because a confidently wrong forecast is what produces a large stake on a losing bet. Log loss for a naive 50% forecaster on a binary event is ln(2) ≈ 0.693.
Calibration: the check that scores cannot replace
Scores are scalars; they compress everything into one number and can hide compensating errors. Calibration exposes structure. Bucket every forecast by stated probability, then compare the bucket's midpoint against the observed frequency inside it:
| Stated probability | Forecasts in bucket | Observed frequency | Reading |
|---|---|---|---|
| 50–60% | N₁ | should be ≈ 55% | above → underconfident |
| 60–70% | N₂ | should be ≈ 65% | below → overconfident |
| 70–80% | N₃ | should be ≈ 75% | the classic overconfidence zone |
Plotted, this is a reliability diagram: perfect calibration is the diagonal, and a curve sagging below it is a model that believes itself more than the evidence supports. Overconfidence is the standard failure mode, and it is invisible to hit rate and nearly invisible to a single Brier number.
The Murphy decomposition splits Brier into reliability (calibration error), resolution (ability to separate events that happen from those that do not) and uncertainty (inherent to the events). A model can be perfectly calibrated and useless — always predicting the base rate — so calibration must be read alongside resolution.
The benchmark that makes any of it meaningful
A Brier score of 0.21 is neither good nor bad on its own. Compute the same score for the de-vigged bookmaker closing price on the same events. If your model does not beat that, it has not demonstrated an edge over the market — however elegant it is.
This is the comparison almost nobody in the prediction-service category publishes. Two exceptions we found: one site displays a single average log loss on its home page, another shows its log loss against the sportsbook's per game. Neither breaks results into probability buckets, so neither shows a calibration curve.
What does CONSENSUS publish, and what does it not?
We publish the settled record, the running result in units, the maximum drawdown, the longest losing run, the sample size and the period — all recountable from the open log.
We do not publish Brier, log loss or a calibration curve today. The reason is sample size: bucketed calibration needs several hundred forecasts per bucket before the observed frequency stops being noise, and roughly a hundred settled signals in total does not support it. Publishing a reliability diagram on this sample would produce a chart that looks scientific and means nothing — the exact failure this guide exists to describe.
Check any of this against our record
Every signal CONSENSUS publishes carries the bookmaker odds fixed before the event starts and the settled result afterwards — including the drawdowns and the losing runs. The running total is on the front page and every entry is in the log.