Why are people afraid of ghosts?

To me the idea of being alone at night in a dark "haunted" place like old house, old castle, or old prison is scary and the idea of seeing or hearing a spirit, or ghost as some would say, is even more terrifying. But it doesnt seem logical to fear these things. Even if ghosts are real they're not going to hurt you so why be fearful of them? Is it because we fear the unknown?