I wish PFF were reliable. I really wish we had an objective grading system that was accurate - taking into account schemes and techniques in addition to end result. One that was fairly administered across the board without human bias and judgement. It's asking a lot. PFF does its best to try to be that. But trying your best doesn't always equate to completing the objective, just ask Nick Hayden. The end result is the grades and resulting rankings are still pretty arbitrary.
"The grading system was founded on the principle of grading “production” rather than traits or measurables, but perhaps a better way to describe it is a player’s “contribution to production” on a given play." (from PFF)
This sounds more like a eye test and with measurables that can change weekly and are graded by different people and are known to the graders only. Not sure at this point how I feel about PFF.
PFF employs over 600 full or part-time analysts, but less than 10% of analysts are trained to the level that they can grade plays. Only the top two to three percent of analysts are on the team of “senior analysts” in charge of finalizing each grade after review. Our graders have been training for months, and sometimes years, in order to learn, understand and show mastery of our process that includes our 300-page training manual and video playbook. We have analysts from all walks of life, including former players, coaches and scouts. We don’t care if you played.
Each grade is reviewed at least once, and usually multiple times, using every camera angle available, including All-22 coaches’ tape.
I know Jordan Palmer ex QB does PFF grading. They stick to a simple system and every player is graded exactly the same.