I'm not saying it's impossible have visitors from other worlds, but there are so many obstacles to overcome, not the least of which is time. The closest star system to ours is Alpha Centauri, a binary system, which makes it more difficult to find the conditions to support life. Alpha Centauri is more than 4 light years away. If there were intelligent life there, it would take an extremely long time to get here, if they decided this is where they wanted to go. Unless they could approach light speed, it would probably take generations of travel to reach us...to hide from us? If they figured out a way to exceed light speed, without becoming infinitely large, or figured out how to manipulate and stabilize wormholes, or somehow take advantage of other dimensions, then maybe they've come here.....but again, why hide from us?
Any sort of fuel or raw materials could be had in abundance from gas giants, nebulae, asteroids, etc. They certainly wouldn't need to come here to strip our planet of its resources.
I think it would make more sense to assume that if we're being visited, it's by our future generations, for whatever reason.