NewsBot
New Member
- Messages
- 111,281
- Reaction score
- 2,947
A look at preseason win projections for the last two years shows that they deliver better results than a coin flip. But not by much.
Few things get football fans more riled up at this time of year than win-total projections for the coming season, and the reaction to those projections is completely predictable.
In that regard, Cowboys fans are no different from fans of all other teams. One part of the fan base is outraged that anybody could think the Cowboys could improve versus last year’s 9-7 team, another part is equally outraged that anybody could think the Cowboys could regress versus last year, a third part can’t understand how anybody could see much of a change in either direction.
But is getting worked up over those projections really worth it?
“Of course it is!” says the fan in me, “not so fast” says the nerd in me.
Brian Burke, formerly of Advanced Football Analytics and now working for ESPN, published an article in 2007 titled ‘Pre-Season Predictions Are Worthless’ and followed that up in 2010 with an article appropriately titled ‘Pre-Season Predictions Are Still Worthless’. Today I’m going to nerd out and use Burke’s methodology to understand whether anything has changed versus his assessments from 2007 and 2010.
Like Burke, I’ll use the Vegas and Football Outsiders projections as the consensus ‘expert’ projections, and add the ELO-based projections from 538.com to the mix. You can look up the Vegas projections for 2016 and for 2017 by following the links, same thing for the 538.com projections from 2016 and 2017, the Football Outsiders projections are taken from the FO Almanac they publish annually.
To measure the accuracy of the projections, Burke used a metric called mean absolute error (MAE). This is the average of the absolute difference between the projection and the actual result. If the MAE for a projection is 3.0, it would mean the projection was off by an average of 3.0 games in either direction. Obviously, the smaller the MAE, the better the prediction.
Last year, Football Outsiders delivered the best MAE of the three expert projections. FO had an MAE of 2.2, meaning their projections were off by an average of just 2.2 wins for each team. The Vegas projections came in at a 2.4 MAE in 2017, as did 538. For a sport that only has a 16-game season, that sounds like a lot. But is it really?
To judge how good those values are, Burke proposed three very simple benchmarks to compare the results against, and I added a fourth:
1. Last Year: This simply uses the previous year’s win total as the projection for the current year.
2. The Coma Patient: Burke calls this the Constant Median Approximation system, or CoMA for short. This is a mindless 8-win prediction for every team. Parity rulez!
3. Regression To The Mean: The assumption here is that teams tend to return towards a .500 record. Teams with a winning record tend to decline; teams with a losing record tend to improve. The formula we’ll use to calculate this is straightforward: Starting with 8 wins, add 0.25 wins for every win above 8 from the previous year, or subtract 0.25 wins for every win less than 8 from the previous year. For example, a 12-win team from 2016 would be predicted to have 8 + (12-8)/4 = 9 wins in 2017.
4. Pythagorean Formula: Also known as the “Pythagorean expectation”, this formula translates a team’s points scored and points allowed into an “expected” winning percentage (for more details, go here). For our exercise, we’ll use the expected wins from each season as the win projection for the next season.
Here’s how the expert projections compare to those benchmarks over the last two seasons.
Last Year
Coma
Patient
Patient
Regression
to the mean
to the mean
Pythagoren
Formula
Formula
538.com
Vegas
Football
Outsiders
2016 Outsiders
2.9
2.5
2.5
2.6
2.4
2.3
2.3
2017 3.1
2.6
2.5
2.3
2.4
2.4
2.2
The expert projections come out ahead in this comparison, but not by much. And as Burke showed, that hasn’t always been the case in the past, though FO did have an impressive performance last year. Still, when some fairly simple math produces almost the same level of accuracy as the experts do, that raises some questions about the validity of the expert projections.
But don’t be fooled by these numbers and assume that FO and Vegas don’t know what they are talking about. If you’re so inclined, and think 2.2 is an entirely unimpressive number, try coming up with an algorithm that delivers better results (the cumulative average error across all 32 teams for the 2017 projection was 70.9 for FO and 73.7 for the Pythagorean Formula). I promise you, that’s going to be very, very tough.
How do you account for a team like the Giants dropping by eight wins from 11-5 in 2016 to 3-13 in 2017? Or the Rams and Jaguars both improving by seven wins? Or the Browns dropping from 1-15 to 0-16?
The NFL is a closed system in which teams tend toward a .500 record, and an 8-8 record is mathematically the most likely outcome. In this system, every win above or below eight wins is increasingly less likely, and projecting a team for an 11-5 or 12-4 record is a risky move - or a bold move, depending on your point of view.
Yet many experts (FO, 538, and Vegas are just the proxies used in this article for those experts) regularly pick teams to have very good or very bad records. Some of these experts use complex mathematical models, some use their expertise, many use a combination of both (some just hold a finger to the wind - but that’s a topic for a different day). And the fact that the experts beat an algorithm based on the inherent parity of the league shows that their models and/or expertise are worth at least something.
But how much? That’s hard to say. Predicting the Patriots for a double-digit win total in 2018 doesn’t seem like much of a stretch, given that they’ve had 15 consecutive seasons with double-digit wins. But consider that outside of the Patriots, there are only three other teams in the NFL who’ve had double-digit wins in both of the last two seasons.
All other teams are yo-yo teams that bounce up and down but haven’t shown the ability to win consistently. You might be high on the likes of the Eagles (13-3 last year), Saints (11-5), Rams (11-5), or Jaguars (10-6) for 2018. But the odds that they’ll repeat as double-digit winners are slim.
Think of the NFL as a game of coin flip, where you get to flip the coin 16 times. If you had to put money on the outcome, the best bet would be an 8-8 result. Expecting a 12-4 result, while statistically possible, would not be very probable.
For Cowboys fans, there is an upside to all of this. Vegas is projecting Dallas as an 8.5-win team, FO are projecting Dallas for 9.1 wins. Assuming another MAE of around 2.3 this year, the Cowboys are projected anywhere from 6.2 to 11.4 wins in 2014. So sayeth the experts.
There should be a result somewhere in that range that should suit just about every Cowboys fan.
Continue reading...