i don't know about "must see," but here are some non-Broadway shows that are either on MY personal "to see" list, or I've already seen them and liked them:
--Twelfth Night (Classical Theatre of Harlem) - starring recent Tony nominee, Kara Young. She's a real star in the making, and this a great role for her.
--The Nosebleed (LCT3)
--The Seagull (Elevator Repair Service)
--Heart (Audible Theatre)