Fans place too much emphasis on college production, for one. It's a useful data point to see a player at a big program outperform NFL-calibre players at the college level since, obviously, if a player can't clear that hurdle he's unlikely to clear the bigger hurdle of the NFL.
With smaller school programs, scouts don't often have the benefit of seeing the best small-school players compete against the best college competition. Lacking that data, small school prospects aren't as sure a bet as vetted large school prospects and they go lower in the draft as a result since the draft is largely an exercise in avoiding failure. It's a misconception that these players necessarily have less physical or intellectual ability. Once they're acclimated to the professional playbook, they're on pretty close to an equal footing. I'm sure there are a number of small school players not in the NFL who could have developed into stars given the right circumstances and opportunities.
With as much money as the NFL spends in scouting each year, the role of college scouts in selecting which players they place in the national spotlight has a lot to do with the makeup of the NFL's rosters. Obviously, the college guys do a good job, too, but it's not surprising that players fall through the cracks.