How Passer Rating Works

[erod]

The ESPN quarterback ratings are far better. It takes into account passes thrown away on purpose, and doesn't credit garbage time stats as much. It also takes into account a swing pass in the backfield that goes 90 yards for a score, versus a bomb hitting a receiver in stride.

 

[percyhoward]

Percy - you cannot take a single example and claim it proves your point.

I'm looking for evidence of where this substantial mathematical difference that you're talking about manifests itself in the practical world of the rankings.

Again, can you show this evidence?

 

[AbeBeta]

I'm looking for evidence of where this substantial mathematical difference that you're talking about manifests itself in the practical world of the rankings.

Again, can you show this evidence?

Evidence. Your claim that a correlation of .51 and .55 is meaningless and that they are the "essentially the same statistic'' is totally flawed and without basis. This tells us that over the time period you cite, PR was CONSISTENTLY less accurate in predicting WINS. You provide no argument based on WINS.

I don't need to show you evidence. It is right there in the values you cite. That is, if you have more than an Intro Stats understanding of what those values mean.

 

[theogt]

Evidence. Your claim that a correlation of .51 and .55 is meaningless and that they are the "essentially the same statistic'' is totally flawed and without basis. This tells us that over the time period you cite, PR was CONSISTENTLY less accurate in predicting WINS. You provide no argument based on WINS.

I don't need to show you evidence. It is right there in the values you cite. That is, if you have more than an Intro Stats understanding of what those values mean.

Well, I don't think there's any meaningful difference between .51 and .55 correlation coefficients, but I'd point out two things:

1. The other formula is not much different from passer rating. The biggest difference is the inclusion of sack yards, which -- if itself correlates with winning to any real degree -- would presumably add to the correlation with winning percentage if combined with passer rating.

2. Simply having a higher correlation with winning percentage is not the end goal. There are tons more stats that have a higher correlation with winning percentage. The question is -- are those a better judge of individual QB performance?

I don't think there's any real argument for adding sack yardage to formula that ranks the performance quarterbacks.

 

[percyhoward]

Evidence. Your claim that a correlation of .51 and .55 is meaningless and that they are the "essentially the same statistic'' is totally flawed and without basis. This tells us that over the time period you cite, PR was CONSISTENTLY less accurate in predicting WINS. You provide no argument based on WINS.

I don't need to show you evidence. It is right there in the values you cite. That is, if you have more than an Intro Stats understanding of what those values mean.

Abe, I'm not claiming that the difference in the coefficients is meaningless. I'm saying it doesn't significantly affect the rankings to the point where there's any obvious advantage in using one stat or the other. That's the whole purpose of the stats anyway, to see where certain players or team rank, either from week to week, year to year, or against the rest of the league.

When comparing the top teams' rankings in passer rating differential to their rankings in ANY/A differential, you get two strikingly similar lists. When doing the same thing for one team over time with Dallas' offensive and defensive passer ratings and corresponding ANY/A, you get two strikingly similar lists.

So why argue over something that has virtually no effect on the rankings?

 

[Yakuza Rich]

Well yeah, Rich that would be that one year that I said Miller only played 8 games. 1992 isn't "the early 90's" any more than 8 games are 16.

It's saying that for 1992 that the Chris Miller was as effective of a passer of the ball as Aikman was.

This is kinda the entire point of the post, how QB rating works and its level of importance in a season. My counter is that the formula is flawed towards dink-n-dunk passing schemes that throw the ball quite often near the goal line. Therefore, somebody like Aikman has a more difficult time achieving a higher QB rating since he was playing in a deep passing scheme that ran the ball near the goal line compared to Miller in the 'run-n-shoot' which threw a lot of short passes and threw the ball quite a bit near the goal line.

Do I think Chris Miller was a better QB in 1992 than Aikman?

Absolutely not.





YR

 

[percyhoward]

Simply having a higher correlation with winning percentage is not the end goal. There are tons more stats that have a higher correlation with winning percentage. The question is -- are those a better judge of individual QB performance?

Yes. WPA, EPA, and various versions of a ANY/A (and scoring, for that matter) all have a higher correlation with winning, but that doesn't make them passer ratings.

 

[percyhoward]

My counter is that the formula is flawed towards dink-n-dunk passing schemes that throw the ball quite often near the goal line. Therefore, somebody like Aikman has a more difficult time achieving a higher QB rating since he was playing in a deep passing scheme that ran the ball near the goal line compared to Miller in the 'run-n-shoot' which threw a lot of short passes and threw the ball quite a bit near the goal line.

Again, Aikman destroyed Miller in passer rating when you compare each player's five best seasons, so Miller probably isn't the best example to use.

You can only point to one season (1992) in which Miller was anywhere near Aikman's league, and there sample size is a major issue because Miller only played 8 games.

Which is kinda what I said before.

 

[Yakuza Rich]

The issue here is we are missing out on several factors.

One being is that the baselines are skewed and not on the same wavelength because there is nothing in place that puts in strength of schedule.

If a QB averages an 85 QB rating going against defenses that average 70 QB rating allowed, he is a more effective QB than the one that has a 95 QB rating and has faced defenses averaging 100 QB rating allowed.

This is really the biggie with any type of Bill James/Moneyball analysis. How the player performs with relation to the average as the baseline and the environment they are in (be it the opponent, weather, stadium influences, etc).

I don't hate QB rating. And like I said, I find QBRD to be very helpful. But, there are inherent flaws to QB Rating and once we understand that we can understand how to use it for more accurate forecasting.



YR

 

[Yakuza Rich]

Again, Aikman destroyed Miller in passer rating when you compare each player's five best seasons, so Miller probably isn't the best example to use.

You can only point to one season (1992) in which Miller was anywhere near Aikman's league, and there sample size is a major issue because Miller only played 8 games.

Which is kinda what I said before.

Again, the entire point was how important QB Rating is in a season. You can argue Miller's sample size, but if we look at 1995 when the Cowboys won another Super Bowl, there was Scott Mitchell with virtually the same QB rating as Aikman.

My main point is that the metric is flawed for a few reasons, one of them being it favors certain schemes and passing philosophies over others. It's not a useless metric, but it certainly has some noticeable flaws.



YR

 

[percyhoward]

strength of schedule.

Not usually as much of a factor when comparing career numbers, but for week-to-week, and year-to-year comparisons, it makes a difference.

Top 10 Passers 2012 (Rank of Schedule)
1. Rodgers (7th most difficult)
2. Manning (25th)
3. Griffin (19th)
4. Wilson (12th)
5. Ryan (32nd)
6. Brady (15th)
7. Roethlisberger (27th)
8. Brees (9th)
9. Schaub (13th)
10. Romo (6th)

 

[AbeBeta]

Well, I don't think there's any meaningful difference between .51 and .55 correlation coefficients, but I'd point out two things:

1. The other formula is not much different from passer rating. The biggest difference is the inclusion of sack yards, which -- if itself correlates with winning to any real degree -- would presumably add to the correlation with winning percentage if combined with passer rating.

2. Simply having a higher correlation with winning percentage is not the end goal. There are tons more stats that have a higher correlation with winning percentage. The question is -- are those a better judge of individual QB performance?

I don't think there's any real argument for adding sack yardage to formula that ranks the performance quarterbacks.

First - the difference between .51 and .55 is substantial. Anything that improves prediction of a complex outcome is useful.

Second -- I initially thought sack yards was a strange addition on its own. If you add that, QB fumbles, and maybe some QB rushing metric you get a more well rounded picture of a QB's performance IMO. For example, a QB who is really good at getting rid of the ball when pressured is penalized under the PR formula but does get a bounce for avoiding sacks.

I don't understand how not having a higher correlation to winning is not the goal. How your performance relates to the undisputed most important outcome is very important. Unless we are talking about predicting how many points an offense scores, I don't see a better value to evaluate against.

 

[percyhoward]

Again, the entire point was how important QB Rating is in a season.

Ah, OK, you're saying don't put a lot of faith in one season's worth of games. Yeah, it's significant that the skewing you're talking about doesn't seem to withstand larger sample sizes. When you look at the list of QB who have finished in the top 10 in passer rating six times or more, you don't see any Chris Miller's or Scott Mitchell's.

 

[AbeBeta]

Abe, I'm not claiming that the difference in the coefficients is meaningless. I'm saying it doesn't significantly affect the rankings to the point where there's any obvious advantage in using one stat or the other. That's the whole purpose of the stats anyway, to see where certain players or team rank, either from week to week, year to year, or against the rest of the league.

When comparing the top teams' rankings in passer rating differential to their rankings in ANY/A differential, you get two strikingly similar lists. When doing the same thing for one team over time with Dallas' offensive and defensive passer ratings and corresponding ANY/A, you get two strikingly similar lists.

So why argue over something that has virtually no effect on the rankings?

Percival, you are taking one analysis and then applying it in a manner that does not reflect the original analysis. The original analysis address how those values relate to wins. Not how they relate to where a team ranks. Two variables can RANK in exactly the same manner but capture remarkably different amounts of explanatory power. That ANY/A and PR reflect similar rankings fails to provide evidence that the two statistics are indistinguishable.

 

[85Cowboy85]

One point I would make is the "highest correlation with winning" is not enough on it's own. For instance I could award one million passer points whenever the quarterback throws a TD in the last 30 seconds to pull ahead. But this would not be an accurate formula to judge a quarterback skill as he is essentially being rewarded or punished for factors beyond his control. The key to creating any sort of statistic to evaluate a player is to isolate only the factors he does control. Therefore any useful statistic must also be context neutral.

For instance if you know anything about baseball sabermetrics DIPS are defense independent pitching statistics. The idea is this: take the number of flyballs, groundballs, strikeouts and walks and come up with a number which represents the pitcher's true skill level. The idea is to isolate the pitcher from his defense and luck related to balls in play finding holes (the opponents batting average on balls in play was found to be largely uncontrollable by pitchers). The advantage is the contribution of the pitcher was isolated much more so then with ERA which can be strongly influenced by luck or bad defense (not to mention W-L record). Thus they are considered by the sabermetric community to be the best indicator of a pitchers skill.

On the other hand in football we are going backwards. The latest offerings like ESPN's QBR instead try to take into account a quarterbacks "clutch" rating. Which as has been discussed before is not context neutral. Instead they should look to further isolate the quarterbacks performance perhaps accounting for things like skill of receivers, time in the pocket, strength of competition ect. I don't deny the passer rating is imperfect but most attempts to fix it actually are worse then passer rating itself.

 

[AbeBeta]

One point I would make is the "highest correlation with winning" is not enough on it's own. For instance I could award one million passer points whenever the quarterback throws a TD in the last 30 seconds to pull ahead. But this would not be an accurate formula to judge a quarterback skill as he is essentially being rewarded or punished for factors beyond his control. The key to creating any sort of statistic to evaluate a player is to isolate only the factors he does control. Therefore any useful statistic must also be context neutral.

For instance if you know anything about baseball sabermetrics DIPS are defense independent pitching statistics. The idea is this: take the number of flyballs, groundballs, strikeouts and walks and come up with a number which represents the pitcher's true skill level. The idea is to isolate the pitcher from his defense and luck related to balls in play finding holes (the opponents batting average on balls in play was found to be largely uncontrollable by pitchers). The advantage is the contribution of the pitcher was isolated much more so then with ERA which can be strongly influenced by luck or bad defense (not to mention W-L record). Thus they are considered by the sabermetric community to be the best indicator of a pitchers skill.

On the other hand in football we are going backwards. The latest offerings like ESPN's QBR instead try to take into account a quarterbacks "clutch" rating. Which as has been discussed before is not context neutral. Instead they should look to further isolate the quarterbacks performance perhaps accounting for things like skill of receivers, time in the pocket, strength of competition ect. I don't deny the passer rating is imperfect but most attempts to fix it actually are worse then passer rating itself.

Poppycock.

This isn't fantasy football.

In the NFL winning is the only outcome that matters.

 

[percyhoward]

Two variables can RANK in exactly the same manner but capture remarkably different amounts of explanatory power.

Yes, that coincidence can indeed occur.

Now show how my examples were instances of such a coincidence.

 

[AbeBeta]

Yes, that coincidence can indeed occur.

Now show how my examples were instances of such a coincidence.

A. That is not a coincidence.

B. Your "examples" have nothing to do with the winning % analysis. You are saying that ANY/A and PR both predict offense rank similarly. That tells us nothing about the winning % which is the analysis you told us showed no difference.

C. (Bonus!) Here are three sets of #s A(1, 2, 3), B(1, 2, 40), C(0, 0, 1) -- let's say C is wins -- you get a 0 for a loss a 1 for a win. A and B could be two different passer ratings. Both A and B produce the same ranking (first is lowest, second middle, last highest). However the correlations of each with wins are different, meaning one is more predictive. Simple statistical fact there.

 

[85Cowboy85]

Poppycock.

This isn't fantasy football.

In the NFL winning is the only outcome that matters.

Winning also matters in baseball or any other sport. The idea is to isolate the factors purely under the quarterbacks control which lead to winning. That's what leads to a statistic which can evaluate a player accurately.

 

[AbeBeta]

Winning also matters in baseball or any other sport. The idea is to isolate the factors purely under the quarterbacks control which lead to winning. That's what leads to a statistic which can evaluate a player accurately.

There are no factors "purely under the quarterbacks [sic] control."