"Isn't Harry's reason for leaving (and deliberately ruining the miracle) because of his anger and jealousy over Jean's wedding to Charlie? I haven't seen the show for a while, but does he actually say elsewhere that he doesn't believe in the town? (Please correct me if I'm wrong.)"
Yes, that is his specific reason. But what I meant is, I would assume there would be LOTS of specific reasons why lots of people in the village might want to leave eventually -- to go and study somewhere, as Charlie Dalrymple has done (he has just returned); to go and live in a bigger city somewhere; and so on and so on and so on. When you think it about it, BRIGADOON is sort of a conservative fable about people who want to live their whole lives in the place where they were born and who don't want ANY changes to their lifestyle in terms of social or technological progress. (I guess somewhat similar to the Amish?) And while that may be what SOME people want, I think it's really pushing it to believe that everyone, or even the majority of people, in the village would feel that way.