This is where my doubt comes from:
Tywin Straight-Up Says That Tyrion Isn’t His Son
Although he raises Tyrion alongside Cersei and Jaime (which many think he only does because he can't bear to destroy something his beloved wife created), Tywin makes his distaste for Tyrion plain on a number of occasions, especially in a scene in the season three episode "Valar Dohaeris." "You are an ill-made, spiteful little creature full of envy, lust, and low cunning," he tells Tyrion. "Men's laws give you the right to bear my name and display my colours since I cannot prove that you are not mine." Taken literally, that last line seems like an obvious clue that Tywin is well-aware that Tyrion is not entirely a Lannister.
At the end of season four, Tywin brings Tyrion's lineage into question yet again when he says "You're no son of mine," just after Tyrion shoots him in the stomach with an arrow. To be fair, Tywin is probably only saying that given the circumstance (read: bleeding out on a toilet), but it could also be his final declaration of Tyrion's illegitimacy.
