How to judge whether an FPL prediction model is any good

Why point predictions look bad by design

FPL scores are lumpy: a huge mass at 1 and 2 points, then jumps at 5, 6, 8 and above. No projection of 4.8 will ever match an actual score of 2 or 9 exactly, so headline error looks terrible even for a good model.

That is expected. The purpose of a projection is to rank options correctly and price risk, not to hit the exact score.

The four metrics that matter

• —Mean absolute error, overall and by position — the raw distance between projected and actual points
• —Minutes error — average absolute difference between projected and actual minutes, the root cause of most points error
• —Start accuracy — the share of starters correctly predicted, which should be well above 85 per cent for a model with a maintained depth chart
• —Captain efficiency — recommended captain's actual points as a share of the best possible captain that week

Calibration beats accuracy

A better question than "how close is the average" is "are the probabilities honest". Of all players the model gave a 70 per cent chance of playing 60 minutes, did roughly 70 per cent do so? Of the outcomes it placed between the 10th and 90th percentile, did about 80 per cent land inside?

A well-calibrated model with modest accuracy is far more useful than an accurate-looking model whose stated confidence is fiction, because you can size your risk against it.

Beware the backtest that cheats

Retro-fitting a model to a completed season using information that was not available at the time — final prices, end-of-season minutes, known injury outcomes — produces spectacular numbers and no predictive power. A valid backtest reconstructs the exact information state before each deadline.

The honest version of this is a forward-tested record: the model publishes its projections before the gameweek, and its error afterwards, permanently.

What FPLHERMES publishes

After every completed gameweek the engine scores itself and writes the results to a public audit page: error overall and by position, minutes error, start accuracy and captain efficiency, for every gameweek including the ones it got wrong. If a tool will not show you a bad week, assume there have been several.