An excerpt from Rule 12: Article 3 that Marcus posted in a different thread -
(c) Uses his hands or arms to materially restrict or alter the defender’s path or angle of pursuit. It is a foul regardless of whether
the blocker’s hands are inside or outside the frame of the defender’s body. Material restrictions include but are not limited to:
(1) grabbing or tackling an opponent;
(2) hooking, jerking, twisting, or turning him; or
(3) pulling him to the ground.
Penalty: For holding by the offense: Loss of 10 yards.
Blocking Notes:
(1) When a defensive player is held by an offensive player during the following situations, Offensive Holding will not be
called:
(a) if the runner is being tackled simultaneously by any defensive player;
(b) if the runner simultaneously goes out of bounds;
(c) if a Fair Catch is made simultaneously;
(d) if the action clearly occurs after a forward pass has been thrown to a receiver beyond the line of scrimmage;
(e) if the action occurs away from the point of attack and not within close-line play;
(f) if a free kick results in a touchback;
(g) if a scrimmage kick simultaneously becomes a touchback;
(h) if the action is part of a double-team block, unless the defender splits the double team, gets to the outside of either
blocker, or is taken to the ground; or
(i) if, during a defensive charge, a defensive player uses a “rip” technique that puts an offensive player in a position
that would normally be holding.
Exception: Holding will be called if the defender’s feet are taken away from him by the blocker’s action.
Notice the part I highlighted. That would seem to indicate that it's not, in fact, legal to grab the jersey or pads, even in the chest area, as many have stated it is.