I don't think size is the issue. There are other starting running backs his size who do perfectly fine. I do think role is the issue. I think Pollard is a role-playing back who we are treating like Ezekiel Elliott, requiring a bunch of middle runs to tighten up the defense when he needs to run outside-in. I don't know if durability is an issue, but certainly when he got more carries last year, his body didn't hold up to it.
His numbers last year (and Elliott's numbers) showed that Pollard deserved more of a chance to carry the ball, but there is a difference in what is asked of the starter and what is asked of the change of pace. Fans (and front offices, too) fall into the trap of believing what a player accomplishes in a backup/change-of-pace role will translate to what he accomplishes in a starter role. Sometimes it does, but often it does not. He's still averaging 4.2 per carry and anyone who was expecting him to average 5.2 like he did last year when he was sharing the load wasn't being realistic. Football is a complementary game, and we have no complement to let him do what he does best.