News: BTB: 2016 NFL Win Projections: Are They Better Than A Coin Flip?

[NewsBot]

A look at pre-season win projections for the last four years shows that they deliver better results than a coin flip. But not by much.

On the day the 2015 season kicked off, Will Brinson of CBS Sports reviewed the 2016 Super Bowl odds for all 32 NFL teams. At the time, the Seahawks (9-2 odds), Packers (6-1), Eagles (8-1), Cowboys (14-1), Cardinals (30-1), and Lions (30-1) were projected as the top NFC teams. Only three of those six teams made the playoffs and only the Cardinals won their division.

It's a testament to the unpredictable nature of the NFL that preseason favorites fall from grace due to key injuries (Cowboys), loss of key players in free agency (Lions), or because they're full of hot air (Philly). In their stead, a new crop of teams sets out every year to re-confirm the old adage that the playoff field churns by about 50% from year to year.

Yet knowing all of that, few things get football fans more riled up at this time of year than win-total projections for the coming season. Earlier this month, our own Dave Halprin published an article looking at the Preliminary Win Projections for the 2016 season by Football Outsiders. A little before that, we looked at the early win projections from the Vegas bookmakers.

And the reaction of Cowboys fans to both sets of projections was as predictable as it's always been. One part of the fan base is outraged that anybody could think the Cowboys will have a winning season in 2016; another part is outraged that anybody could think the Cowboys could have a losing season; a third part can't understand what the fuss is all about.

I will admit it's kind of fun to be accused of being a total Cowboys homer after years of being accused of irrational Cowboys hatred.

— Aaron Schatz (@FO_ASchatz) May 13, 2016​

But is getting worked up over those projections really worth it? Brian Burke of Advanced Football Analytics published an article in 2007 titled 'Pre-Season Predictions Are Worthless' and followed that up in 2010 with an article appropriately titled 'Pre-Season Predictions Are Still Worthless'. Today I'm going to use Brian's methodology to understand whether anything has changed versus his assessments from 2007 and 2010.

Like Burke, I'll use the Vegas and the Football Outsiders projections as the consensus 'expert' projections. You can look up the Vegas projections here (2012, 2013, 2014, 2015), the 2012-13 Football Outsiders projections are taken from the FO Almanac they publish annually, the last two years are taken from their final DVOA projections (2014, 2015).

To measure the accuracy of the projections, Burke used a metric called mean absolute error (MAE). This is the average of the absolute deviation between the predicted and the actual win total. If the MAE for a projection is 3.0, it would mean the projections were off by an average of 3.0 games in either direction. Obviously, the smaller the MAE, the better the prediction.

Over the past four years both Football Outsiders and the Vegas betting lines have achieved an MAE of 2.3 pretty consistently (Vegas hit 2.0 once in 2014), meaning their projections were off by an average of 2.3 wins for each team. For a sport that only has a 16-game season, that sounds like a lot. But is it really?

To judge how good those values are, Burke proposed three very simple benchmarks to compare the results against:

Previous Year: This simply uses the previous year's win total as the projection for the current year.

The Coma Patient: Burke calls this the Constant Median Approximation system, or CoMA for short. This is a mindless 8-win prediction for every team.

Regression To The Mean: The assumption here is that teams tend to return towards a .500 record. Teams with a winning record tend to decline; teams with a losing record tend to improve. The formula we'll use to calculate this is straightforward: Starting with 8 wins, add 0.25 wins for every win above 8 from the previous year, or subtract 0.25 wins for every win below 8 from the previous year. For example, a 12-win team from 2013 would be predicted to have 8 + (12-8)/4 = 9 wins in 2014.

Here's how Vegas and Football Outsiders compare to those benchmarks:

Previous
Year Coma
Patient Regression
to the mean Vegas Football
Outsiders​

2012

3.1 2.6 2.4 2.3 2.3​

2013

2.8 2.5 2.4 2.3 2.3​

2014

2.3 2.7 2.4 2.0 2.3​

2015

2.7 2.5 2.3 2.3 2.3​

Average

2.7 2.6 2.4 2.2 2.3​

​

The Vegas and FO projections come out ahead in this comparison, but not by much. And as Burke showed, that hasn't always been the case in the past. And when a simple regression-to-the-mean algorithm produces almost the same level of accuracy as the experts do, does that raise questions about the validity of the expert projections?

If you're so inclined, and think 2.3 is an entirely unimpressive number, try coming up with an algorithm that delivers better results (The cumulative average error across all 32 teams for the 2015 projection was 72.0 for Vegas and 73.9 for FO). I promise you, that's not going to be easy. I know because I tried.

So don't be fooled by these numbers and assume that FO, Vegas and some other experts don't know what they are talking about. Here are the MAE results for some other win projections for 2015:

538.com: 2.2 average MAE (69.7 cumulative average error)

CBSSports.com: 2.2 (70.0)

ESPN.com: 2.2 (71.0)

USA Today: 2.6 (84.0)

Ultimately though, these result are probably much less about the validity of expert opinions and much more about the folly of NFL predictions as the New York Times explains:

The experts’ inability to convincingly beat mindless guessing strategies doesn’t necessarily mean that they are bad predictors. As Stuart points out, predicting wins and losses in the N.F.L. may just be a hopeless endeavor, even for advanced models.

"I think there is just too much randomness and unpredictability in the N.F.L. from season to season, and too much in-season variance, to make the predictions much more accurate," he said.
​

The NFL is a closed system in which teams tend towards a .500 record, and an 8-8 record is mathematically the most likely outcome. In this system, every win above or below eight wins is increasingly less likely, and projecting a team for an 11-5 or 12-4 record is a risky move - or a bold move, depending on your point of view.

Think of the NFL as a game of coin flip, where you get to flip the coin 16 times. If you had to put money on the outcome, the best bet would be an 8-8 result. Expecting a 12-4 result, while statistically possible, would not be very probable.

Yet many experts regularly pick teams to have very good or very bad records. Some of these experts use complex mathematical models, some use their expertise, many use a combination of both. And the fact that they beat an algorithm based on the inherent parity of the league shows that their models and/or expertise are worth at least something.

How much? That's hard to say. Predicting the Patriots for a double-digit win total in 2015 doesn't seem like much of a stretch, given that they've had 12 successive seasons of double-digit win totals. But that model won't always work. The Ravens had a streak of seven consecutive non-losing seasons with six playoff appearances come to an abrupt end with a 5-win season last year. And don't even start with the Cowboys' 8-game drop from 2014 to 2015.

Predicting very high or very low win totals requires a lot of confidence in your projection methodology, and comes with a Catch-22: If you project very high win totals, you'll be accused of being overconfident. If you don't project high win totals, you'll be accused of not having any confidence in your methodology.

For Cowboys fans, there is an upside to all of this. Vegas is projecting Dallas as a 9-win team, FO are (currently) projecting Dallas for 9.6 wins. Assuming another MAE of 2.3 this year, the Cowboys are projected anywhere from 6.7 to 11.9 wins in 2016. So sayeth the experts.

There should be a result somewhere in that range that should suit just about every Cowboys fan.

Continue reading...