Back around '62 before my family became multi-racial and multi-ethnic, I saw Auntie Mame on TV. I was aware then (even though I was only 13 or so) that the way Ito spoke and acted was not the way Asian guys my age ever talked. I knew it was mean-spirited to portray him that way and painfully unfunny.
My younger brother married Miyung (who was born in Korea) sometime in the 80's. By the mid-90's I had 2 nieces (Linda and Lisa) and a nephew (Kenny) who loved to go through my movie collection and pick out films about people who looked like them. I always hid Auntie Mame and Breakfast at Tiffany's when I knew they were coming over because they were much too young to have their feelings hurt back then. They didn't seem to mind films like (Here we go again!) The King and I, Flower Drum Song, Dragon Seed, The Good Earth, The Teahouse of the August Moon, and Charlie Chan movies (but only the 20th Century Fox series, never the low budget Monogram ones).
Lisa absolutely loved Flower Drum Song. I was amazed that she could delineate the ethnic make-up of all the Asian-American cast members just by looking at them, e.g., she would say "Mei Li, Sammy, Ta, and Helen are Japanese, Linda and Chi-Yang are Chinese, and San is Filipino." To which I would remark skeptically, "C'mon, you must have looked that up, you little cheater. They're all Asian." Her quick reply was always (jokingly, I hope) "Oh Uncle Tony, you are so racist. I just know."