Wait...so you build the viewer...but somehow they can't see you and you can't affect anything. That's completely impossible. Just by observing you're affecting the past. Why? I'm glad you didn't ask but I'm going to answer anyway.
Think about it! What is needed to view something? Photons. So photons have to be able to travel through the viewing device from the past to the present in order for you to see the events in the past. That means you're stealing energy from the past. Photons that might have gone on to affect something else. You're stealing energy from the past!
(Yes, it's a bit tongue in cheek, but you are affecting the overall energy state of the universe, in however insignificant amounts, causing that energy to basically cease to exist for the duration of the time span between what you're viewing and the point from which you are viewing.)