The reflection of the visible spectrum which includes all colors is considered to be white where the absorption of light(all color) is black. Therefore the true people of color are
We're all polka dotted stripped lizard cats for all I care.
By that logic, white people would reject all colors and black people would accept them. Making black people still the people of color.
Black people aren't actually black and white people aren't actually white.
We're all people of color; we're all a spectrum of brown tones. No one is truly black or white.