NewsBot
New Member
- Messages
- 111,281
- Reaction score
- 2,947
With OTAs underway, and reports emerge extolling the play of the Cowboys most recent draft crop, it's important that we take a step back and recalibrate our expectations to the actual historical performance of first-year players. Here, in the fourth installment of a multi-part series, we investigate the draft's fundamental uncertainty.
In the Cowboys' pre-draft press conference, Stephen Jones was asked how the extra preparation time - due to the draft being in early May instead of late April - impacted the team's scouting and evaluation process. Jones the Younger offered that it allowed the coaches and scouts extra time to get on the same page, concluding that "you just get to do more diligence. It only helps." A follow-up question, by Todd Archer, asked whether the extra two weeks might lead to over-analysis. Jones responded firmly:
I don't think so....I haven't seen people backing up on things. I think it just allows you to do a little more further study on guys and make sure you have them right and you haven't overlooked something. I don't think the ‘study long, study wrong' is an issue here. I think it's been good for us and we certainly look at it as a way to be even better than we've been in the past.
That Stephen believes this to be true is all well and good. Evidence suggests, however, that he's wrong. In an article that came out the day before the draft, Joseph Stromberg explored the idea that NFL teams draft players irrationally. To substantiate this, he turned to one of my favorite thinkers about the way teams behave when it comes to the draft, Cade Massey.
Massey is a professor whose area of specialty is the psychology of overconfidence, particularly as it applies to making decisions in uncertain environments. One of his favorite test cases is the NFL Draft; with Richard Thaler, he has published an article, "The Loser's Curse" (hit the link for a .pdf), in which they conclude that, because the cost of players falls off far more quickly than their average performance, teams are better off trading down to late in the first or even into the second round, where they can get more players cheaper who, taken together, will contribute more overall production (note that this is before the new CBA was signed, restoring some equilibrium to the first round).
He followed this up in March 2012 with a presentation at the MIT Sloan Sports Analytics Conference entitled "Flipping Coins in the War Room." In it, he made some startling points, the foremost of which was to dispel the notion that certain teams "draft better" than others. Success in the draft, Massey proffered, is not a matter of skill, but of luck. And not just a bit of luck, but entirely luck. As he surmises, what the statistics are "saying is that there are no differences across teams...literally zero differences across teams in player-picking ability." On a skill-based to chance-based continuum, the ability to draft players falls at the far end of the chance-based side of the spectrum.
Certainly some teams draft better than others. This is not because they are more skilled, Massey argues, but because they do a better job managing chance. A rational being, he notes, will go about the decision-making process differently when solving math problems (pure skill) than when entering the lottery (pure luck). He proposes that successful teams operate according to a "logic of chance," a distinct set of proscriptions for operating in an environment with high uncertainty. This is where teams behave "irrationally" to Stromberg: by treating the draft like a math problem.
According to Massey, the most successful drafting teams will 1) acknowledge the role of chance in the drafting process; 2) minimize the cost of choosing (i.e., don't trade up); and 3) maximize the number of chances (this doesn't mean accumulating 35 seventh-round picks; rather, it means acquiring as many "premium" picks as possible).
Stromberg points out that Massey works as a consultant for several unnamed NFL teams, so he's had the opportunity both to discuss the logic of chance with NFL front offices and to observe how, and to what degree, they incorporate his teachings. And it appears that most fail, largely because they cannot adhere to the logic of chance:
...in my experience, teams always say they're on board with it in January. Then when April rolls around, and they've been preparing for the draft for a long time, they fall in love with players, get more and more confident in their analysis, and fall back into the same patterns.
In short, NFL front offices fall prey to a widely-known psychological bias called the overconfidence effect (recall that Massey's specialty is the psychology of overconfidence), which has been observed everywhere from bettors picking horses to psychologists making nosological diagnoses. Its tenet is simple: as people are given more information, the accuracy of their analysis often hits a ceiling but their confidence in it continues to increase.
Many experts have made their careers by being knowledgeable or being able to make critical distinctions between similar instances. In uncertain environments, however, they struggle to admit that they don't know, and thus are unwilling to acknowledge the role of luck in the process in which they are invested. This is particularly interesting when it comes to the draft, a process where success requires that experts acknowledge the limitations of their expertise.
Thus when somebody like Stephen Jones triumphantly declares that the extra two weeks allowed his team further study ("more diligence"), its good to remind ourselves that this is precisely the situation wherein experts are able to fool themselves into thinking that they know: their analysis has already hit a ceiling, but they give themselves two extra weeks in which to increase their confidence that one side of the coin is better than the other.
And this brings me to the Cowboys' second-round choice, DeMarcus Lawrence. To wit:
- They talked themselves into believing that they had to have Lawrence. As Stromberg and Massey repeatedly point out, a team will often trade up when they identify a player they prefer at a needed position because they're certain one is much better. But this isn't supported by their data: on average, the chance that the first, say, defensive end drafted will start more games than the second one picked is just about a coin flip: 52 percent. Compared to the third DE off the board, it's only 55 percent; compared to the fourth, it's merely 56 percent.
- They traded multiple picks to get to the spot where he was selected. Historically, trading down and getting two players gives the team that trades down an average of five more starts per season and slightly more total Pro Bowls. As Stromberg notes, "The truth is that teams are imperfect talent evaluators, so having two later picks is better than a single early one. Risk diversification at work."
Massey's work makes a strong case that NFL teams cannot make critical distinctions between players who are closely grouped. But they often believe they can, because they fool, study, or talk themselves into a sense of overconfidence that governs decisions like those the Cowboys made in selecting Lawrence. In other words, they ignore the logic of chance, in which there are no right answers, instead seeking a mathematical logic in which a right answer can almost always be found. But the draft doesn't follow mathematical logic.
The larger point here is not to criticize Lawrence or where he was selected in the draft. Rather, it is to reinforce the notion that, when it comes to draft classes, teams incorrectly believe they know how successful the players they have selected will be. And if teams don't know, how could we? With this in mind, I'll remind you of the catchphrase that has governed this series: with this, as with any draft class, it's important to recalibrate our expectations. Otherwise, we fall prey to the same overconfidence in which teams too frequently indulge.
And you know what follows that, right? Disappointment. If the most stalwart pessimist is nothing more than a disappointed optimist, the draft is exactly the place to provide that measure of disappointment.
Continue reading...