Does the setting actually matter for horror?

I've always felt that the best horror relies on a sense of isolation, but it doesn't matter if it's a foggy forest or a lonely island. As long as the atmosphere is thick enough, the specific location is just window dressing, right?