"What actual harm did Jackie Mason do to gays and people of color, for the hate that's being leveled against him now?"
Just one example:
He told jokes making fun of "faygelahs" [his word, not mine, and I have no idea how to spell it] for being who they were. A gay kid's parents went to see his show and came home and told those jokes to the kid's aunts and uncles at the Sunday barbecue. Everyone laughed uproariously. The gay kid killed himself.