Belichick is an excellent coach. Belichick is very conservative when it comes to 4th down decisions. Less conservative than most, but still more conservative than he should be (if winning football games is the only thing that matters). All those things are true at the same time.
Here's the data that is readily available that goes into developing these models:
Success rate on 3rd and 4th down for a given position on the field and yards to go for the first.
Expected points scored for a drive starting at any position on the field.
You can slice and dice that data in various different ways to get the best data set for your team, opponent, etc.
When you use this data to calculate expected results from punting or going for it, it becomes painfully obvious that coaches don't go for it enough. We're not just talking about cases where, well, it's a better-than-average defense, so things go the other way. We're talking about cases where the discrepancies are so huge that no information of that kind is going to change the result. The results are clear, and there's a huge amount of data out there to back it up. Coaches are risk averse.
They're risk averse for a reason. When you go for it on 4th, sometimes you won't make it. That's the nature of risk. And it doesn't matter how often you succeed: the press and the fans and the ownership will point to those times it failed and crucify you for it. Going for it on 4th down much more often will (a) win you more games in the long run, and (b) get people really mad at you whenever it doesn't work. Coaches would rather not deal with the ramifications of (b), so they don't do it.
There are other reasons they don't do it. They really deep down don't believe it, just like so many here don't. Teams should go for it on 4th and 1 from their own 20 yard line (yes, there are certainly game situation exceptions to this). But it feels really wrong to just about everyone, because that's not how the game has ever been played, and because people are by nature risk averse. Heck, it feels wrong to me, even though I know it's right. But this is where "common sense" and "conventional wisdom" and "football 101" fail us. We've seen it in any number of industries. We've seen it in baseball. The conventional wisdom turns out to be wrong when serious analysis is trained on a problem. Not always, by any means, but sometimes.
sorry, maybe I am about to go a little geek on you.........
first, this has nothing to do with fans, press, etc. coaches arent' really concerned about that stuff, that's for the fans and the press and the league....their concern is wins and losses.
there are several things with your hypothesis that are wrong, having done a lot of analysis in my career its quite important to have the right hypothesis and assumption with all the right factors, else you will arrive at wrong conclusions. given you are using statistics and a few vague studies to back up your assumptions lets actually take a look at what you just said...
your initial premise is that the NFL coaches are risk averse and conservative, completely dismissing that perhaps they are smart and through years of experimentation and knowledge they have learned that those risks don't pay....more on that later,
but lets play with your hypothesis, using your own assumption that coaches are conservative. That and that alone makes the stats you provide lopsided and biased, since the statistics for 4th downs point to a time when coaches weren't conservative, they wanted to take risks (which is not their nature as you said) and needed to go for it, in desperate situations, perhaps being too far behind, or no time left, etc. so the stats are not the normal stats of a coach, and all coaches (or most) going for it on a regular basis even in a specific location on the field (in this argument between the 40-mid field of the opponent). the stats are biased because by your own assertion, coaches are conservative and normally don't go for it on 4th down, which makes the 4th down stats biased by your own logic....
what would the stats look like if all or most coaches went for it all the time? do desperate team, have defenses play prevent defenses, where giving up yards, but killing the clock is the right tactic? why do conservative coaches, take risks all of a sudden? you point to those stats where those stats realistically and even scientifically are like a cream soup, with all ingredients (data) mixed in a blender. its all there, but not distinguishable by specifics of the situation, thus leading to incorrect conclusions and in this case, given the coaches are conservative, that data is very relevant to the stats...what would the data look like, if coaches went for it on 4th down all the time.... then again I am a data analyst, what do I know!!!!
but lets forget that argument for a second and look at your stats you used to make your argument...lets continue to play on your hypothesis using your own numbers.
based on what you provided, there is a 50% chance of success, and teams should risk it and go for it, because it leads to more points...great
....but the other side of the coin you conveniently avoided to bring into the discussion, you will fail 50% of the time, thus leading to the other team scoring. you can't assume if I go for it, I make points and the other team getting the ball in great field position and they won't score!!! that would be too biased. then again you need that to make your point.........
and you also have to assume the other team will take the same chances when given short field and on my side of the field, they will take the same risks, go for it, and score 50% of the time, go down the 50% probability of conversion and you will end up
even..... statistically speaking
.........
so if 50% of the time I go for it, I make it and I score. the other 50% of the time I go for it, I don't make it and they score, doesn't that negate the whole thing? and that's what you used to argue that they should go for it, because it leads to scoring and given the same stats you used to argue your point, statisically, it breaks even!!!!
so as a coach, if statistically it breaks even, I don't win, and if my object is to win (more on that later), then I want to increase my chances of winning and statistic (tons and tons of it), show that if the opponent has a longer distance to drive, they have less chance of scoring and provide me with better field position and allow me better opportunity to score...... the stats are there, go check it out for yourself. that's why the collective of coaches, and their years of experience often speak of "field position battle". then again, they are football coaches, what do they know about football and statistics!!!
so going back to your original point, coaches being conservative....perhaps the collective of coaches, over the years, through thousands of games and millions of plays, and thousands of game situations has learned, that being too risky doesn't pay, specially if it breaks things even (btw, we have seen risky coaches come to NFL, trying to establish new way of doing things and they have failed...i.e. Kelly). Furthermore, the Romer paper, based its hypothesis on maximizing profit....yes, makes sense in a corporate wall street world, but in the world of football, which the professor totally missed (given he is a professor with limited knowledge of professional football), its not about maximizing profits, its a "binary" problem. you either win or lose. scores don't really matter. at the end of the day, they don't add up the scores to see who makes it to the playoffs, they add up the wins, and if you have 16 wins by one point, and a team has 9 losses by one point and 7 wins by total of 100 pts, the 7-9 team won't make it to the playoffs. they took chances, they scored and yet they are sitting home watching.....so in a world of binaries, being risk averse is smart, in a world of maximizing profits, taking risks makes sense.
I won't get into the argument of going for it on 4th and 7 as bkight13 initially commented on, GIVEN statiscally speaking based on what you provided, chances of converting and thus scoring are below 50% and in that case, you end up giving up more points, than scoring.....
so there is a reason, coaches like bilicheck, Holmgren, carroll, walsh, landry, McCarthy, Lombardi, etc. are successful, not for being conservative, but by being smart knowing that in football unnecessary risk taking doesn't statistically lead to binary wins and losses.
