If you don't know what you're doing, then, yes, you take the second or third best option available for the snaps you're expecting him to actually play. That way he's ready for all those offensive snaps he's not going to take on gamedays. That makes sense.
The best argument for replacing Brown with a lesser ST player is if you you think the lesser player is actually really close to Brown on STs, is younger and will develop, and if you have control over him contractually longer. But that the bottom of the roster, those guys are just likely to have a paycheck and you're not really worried about losing them at the end of their deals. Really, they'll just go by which one fills the snaps that WR5/6 actually play the best.