"I'd forgotten it's nighttime when they go back ("Laddie, you woke me up!"), and that itself strains credulity. Sure, let's go looking for an invisible village in the pitchblack dark. "I just want to see where it was!" Tommy cries. How can he see where it was if it's the middle of the night?"
Are we sure the script indicates that it's nighttime in Jeff and Tommy's world when they are actually back on the old site of the village? After all, we're told that the villagers go to sleep for 100 years every "night," so that would certainly include 100 years worth of daytimes in the real world :-)