News: BTB: Are NFL Pre-Season Win Projections Worthless?

[NewsBot]

A look at pre-season win projections for the last two years shows that they deliver better results than a coin flip. But not by much.

Few things get football fans more riled up at this time of year than win-total projections for the coming season. On Wednesday, our own Dave Halprin published an article looking at the Preliminary Win Projections for the 2014 season by Football Outsiders. Earlier this week, we looked at the win projections from the Vegas bookmakers.

And the reaction of Cowboys fans to both sets of projections was as predictable as it's always been. One part of the fan base is outraged that anybody could think the Cowboys could improve versus last year; another part is outraged that anybody could think the Cowboys could regress versus last year; a third part can't understand how anybody could see much of a change in either direction.

But is getting worked up over those projections really worth it? Brian Burke of Advanced Football Analytics published an article in 2007 titled 'Pre-Season Predictions Are Worthless' and followed that up in 2010 with an article appropriately titled 'Pre-Season Predictions Are Still Worthless'. Today I'm going to use Brian's methodology to understand whether anything has changed versus his assessments from 2007 and 2010.

Like Burke, I'll use the Vegas and the Football Outsiders projections as the consensus 'expert' projections. You can look up the Vegas projections for 2012 and for 2013 by following the links, the Football Outsiders projections are taken from the FO Almanac they publish annually.

To measure the accuracy of the projections, Burke used a metric called mean absolute error (MAE). This is the average of the absolute value of the error. If the MAE for a projection is 3.0, it would mean the projections were off by an average of 3.0 games in either direction. Obviously, the smaller the MAE, the better the prediction.

For the past two seasons, both Football Outsiders and the Vegas betting lines have achieved an MAE of 2.3 respectively, meaning their projections were off by an average of 2.3 wins for each team. For a sport that only has a 16-game season, that sounds like a lot. But is it really?

To judge how good those values are, Burke proposed three very simple benchmarks to compare the results against:

Last Year: This simply uses the previous year's win total as the projection for the current year.

The Coma Patient: Burke calls this the Constant Median Approximation system, or CoMA for short. This is a mindless 8-win prediction for every team. Parity rulez!

Regression To The Mean: The assumption here is that teams tend to return towards a .500 record. Teams with a winning record tend to decline; teams with a losing record tend to improve. The formula we'll use to calculate this is straightforward: Starting with 8 wins, add 0.25 wins for every win above 8 from the previous year, or subtract 0.25 wins for every win below 8 from the previous year. For example, a 12-win team from 2013 would be predicted to have 8 + (12-8)/4 = 9 wins in 2014.

Here's how Vegas and Football Outsiders compare to those benchmarks:

Last Year Coma
Patient Regression
to the mean Vegas Football
Outsiders​

2012

3.1 2.6 2.4 2.3 2.3​

2013

2.8 2.5 2.4 2.3 2.3​

Average

2.9 2.5 2.4 2.3 2.3​

​

The Vegas and FO projections come out ahead in this comparison, but not by much. And as Burke showed, that hasn't always been the case in the past. And when a simple regression-to-the-mean algorithm produces almost the same level of accuracy as the experts do, that raises a lot of questions about the validity of the expert projections.

If you're so inclined, and think 2.3 is an entirely unimpressive number, try coming up with an algorithm that delivers better results (The cumulative average error across all 32 teams for the 2013 projection was 73.5 for Vegas and 74.1 for FO). I promise you, that's not going to be easy.

So don't be fooled by these numbers and assume that FO and Vegas don't know what they are talking about. The NFL is a closed system in which teams tend towards a .500 record, and an 8-8 record is mathematically the most likely outcome. In this system, every win above or below eight wins is increasingly less likely, and projecting a team for an 11-5 or 12-4 record is a risky move - or a bold move, depending on your point of view.

Think of the NFL as a game of coin flip, where you get to flip the coin 16 times. If you had to put money on the outcome, the best bet would be an 8-8 result. Expecting a 12-4 result, while statistically possible, would not be very probable.

Yet many experts (FO and Vegas are just the proxies used in this article for those experts) regularly pick teams to have very good or very bad records. Some of these experts use complex mathematical models, some use their expertise, many use a combination of both. And the fact that they beat an algorithm based on the inherent parity of the league shows that their models and/or expertise are worth at least something.

How much? That's hard to say. Predicting the Patriots for a double-digit win total in 2014 doesn't seem like much of a stretch, given that they've had 11 successive seasons of double-digit win totals. But the Falcons were in a similar situation headed into the 2013 season - they were coming off three successive seasons with 13, 11, and 13 wins - yet they ended up with just four wins in 2013.

Predicting very high or very low win totals requires a lot of confidence in your projection methodology, and comes with a Catch-22: If you project very high win totals, you'll be accused of being overconfident. If you don't project high win totals, you'll be accused of not having any confidence in your methodology.

For Cowboys fans, there is an upside to all of this. Vegas is projecting Dallas as an 8-win team, FO are (currently) projecting Dallas for 7.1 wins. Assuming another MAE of 2.3 this year, the Cowboys are projected anywhere from 4.8 to 10.3 wins in 2014. So sayeth the experts.

There should be a result somewhere in that range that should suit just about every Cowboys fan.

Continue reading...