If there ever was going to be time travel possible...

There's a bunch of theories involving time-travel only to times where the "time-machine" or contraption existed, which neatly gets around that problem.

It's also likely time-travel would be developed by post-human society, which might have ethical views against messing with primitive societies. The classic SciFi version is just that they would affect their own timeline, the grandfather paradox, and therefore time travel will only be done in a way which won't affect the past. That's kind of lame, but with iron enough grip on the technology, it would probably be possible to hide time-travel until it's officially discovered.