It's not ok to call it racist, because (as I said) it isn't.
rac·ist
/ˈrāsəst/
adjective
showing or feeling discrimination or prejudice against people of other races, or believing that a particular race is superior to another.
Whites are not a minority or marginalized group.
FWIW, based on the little bit I know the "victim" you are worried about, I feel confident he would (or has) laughed at being called this horrible thing.
