The concerns I have about Smith is that in the games I watched, he was pretty accurate on the short stuff, he got rid of the ball quickly enough, and generally did a pretty good job of moving the chains, with one obvious, huge, glaring, monstrous, extremely important, career changing exception. But even when he was killing teams, he wasn't killing them by going deep or hitting guys down the seam, or throwing the 15 or 20 yard outs. He was hitting Ginn and Gonzales on slants, quick hooks, etc.
I don't think his height is near as much of an issue as his ability to make all the throws. But, coincidentally, that's another thing he has in common with Brees. Denver used to kill Brees 2x a year by crowding the short routes and the middle of the field and forcing him to throw deep and outside. And he had a very hard time doing that. That's what Smith is going to have to overcome.
Now, none of us know whether or not Smith is going to be able to that. But I'll go on record saying I don't think too many teams in the NFL are going to spend a 1st day pick on a 6' 1" QBs upside.