Model evaluation · built September 10, 2026
Every settled prediction this project has ever made, scored against the baselines that matter. The model is compared to a coin flip and to simply betting the home team, because a number with nothing to beat is not evidence of anything.
This is a back test, not a track record. only 10 live settled predictions, 250 required before any accuracy figure appears on the site Every number on this page comes from predictions made after the games were played, using data those games had already produced. That is the right way to check whether the maths works and the wrong way to judge whether the model predicts anything. The prospective count is 10 and the page will say so until it reaches 250.
Every input below is cumulative to the day before the game, and every fitted constant is refit from games completed before that date. Nothing here has seen the future.
Point in time inputs, constants refit daily from completed games only.
| Sample | 1309 games, 2026-06-01 to 2026-09-09 |
| Brier | 0.24868 |
| 95% interval | 0.24443 to 0.25294 |
| Coin flip | 0.25000 |
| Skill vs coin | +0.53% |
| Hit rate | 54.2% |
| Calibration in the large | +0.0025 |
| Total MAE | 3.60 against 3.62 naive |
The interval contains 0.25, so this sample cannot distinguish the model from a coin flip.
Constants frozen on the earlier window, then applied unchanged to a later window that was untouched when they were fitted.
| Sample | 543 games, 2026-08-01 to 2026-09-09 |
| Brier | 0.24573 |
| 95% interval | 0.23949 to 0.25197 |
| Coin flip | 0.25000 |
| Skill vs coin | +1.71% |
| Hit rate | 55.2% |
| Calibration in the large | +0.0253 |
| Total MAE | 3.54 against 3.56 naive |
The interval contains 0.25, so this sample cannot distinguish the model from a coin flip.
| Baseline | n | Brier | Log loss | Hit rate |
|---|---|---|---|---|
| coin flip | 1309 | 0.25000 | 0.69315 | 52.5% |
| always the home team at 54% | 1309 | 0.24961 | 0.69238 | 52.5% |
| log5 on season to date win pct | 1309 | 0.25454 | 0.70372 | 53.7% |
The table below is the development back test. It is kept because the gap between it and the walk forward numbers above is the single most useful thing on this page: it shows how much a leaky evaluation flatters a model.
| Version | n | Brier | Log loss | Calib gap | Hit rate | In the large | Total MAE |
|---|---|---|---|---|---|---|---|
| v1.0.0-poisson Poisson, no home field term. The first version. | 544 | 0.24063 | 0.67417 | 0.04629 | 57.2% | +0.0427 | 3.48 |
| v1.1.0-negbin-hfa Negative binomial, home field from the run split. | 539 | 0.24076 | 0.67445 | 0.04211 | 57.7% | +0.0368 | 3.47 |
| v1.2.0-negbin-hfa-logit Negative binomial, home field from the win rate. Current. | 544 | 0.23962 | 0.67209 | 0.02119 | 58.3% | +0.0153 | 3.47 |
| Coin flip every game 50/50 |
544 | 0.25000 | 0.69315 | – | – | – | – |
| Always the home team flat 54.0% |
544 | 0.24822 | – | – | – | – | – |
| Season average total guess the mean every night |
544 | – | – | – | – | – | 3.58 |
Brier 0.23962 against 0.25000 for a coin flip and 0.24822 for always backing the home team. That is a Brier skill score of 4.15% over a coin and 3.46% over home field. Real, measurable, and small. Anyone telling you a baseball model does much better than this over a full season is selling something.
Mean absolute error 3.47 runs against 3.58 for ignoring the model entirely and guessing the season average every night. That is not an edge. The total projection is published because it is an honest output of the model, not because it is good, and it should not be used to price a bet.
The weighted calibration gap is 0.0212 and calibration in the large is +0.0153. The first version was at 0.0463 and +0.0427, which was the model being systematically too low on home teams. Finding and fixing that was worth more than any amount of tuning.
58.3% of games went the way the model leaned. It is on this page for completeness and it is the one figure that should carry the least weight, because it throws away everything the probability was telling you.
| Predicted band | n | Mean predicted | Observed | Gap |
|---|---|---|---|---|
| 30-40% | 21 | 0.366 | 0.381 | +0.015 |
| 40-50% | 167 | 0.459 | 0.449 | -0.009 |
| 50-60% | 276 | 0.548 | 0.573 | +0.024 |
| 60-70% | 78 | 0.636 | 0.667 | +0.031 |
| 70-80% | 2 | 0.725 | 1.000 | +0.275 |
| Version | Reason for the change |
|---|---|
| v1.0.0-poisson | Assumed run totals follow a Poisson distribution and gave the home club no advantage at all. |
| v1.1.0-negbin-hfa | Fixed the distribution (the league variance to mean ratio is 2.32, a Poisson requires 1.0) and added a home term from the measured run split. |
| v1.2.0-negbin-hfa-logit | The run split under-explains the home win rate because the home club does not bat in the bottom of the ninth when it is ahead. Runs stay on the measured split so the score projection stays unbiased; the win probability carries a separate constant fitted to the league home win rate. |
What would make this page dishonest. Quoting the hit rate without the baseline. Quoting the back test as a track record. Changing the model until the back test improves and calling the result a discovery. Dropping the total MAE row because it is unflattering. None of those will appear here.