Of course, plenty of stories (in musicals and otherwise) have similar unexplainable gaffes. One infamous one is - where did Cain and Abel find wives? The Bible never explains that.
I tend to think that we can always just come back to the fact that Brigadoon is entirely fantasy - maybe not the best-constructed fantasy lol, but even so, it's not meant to make perfect sense in reality. For 1947, when musicals were really still beginning to become a more "mature" art form, I think it's still pretty good. Not perfect, but pretty good.