except being labeled either of those (nowadays) does nowhere near the damage, reputational or otherwise, of being labeled a racist.

Only if there were actions to back it up (video, audio, etc.). If it's just someone screaming racist, it's meaningless. I live in Houston where we have Quanell X, the leader of the New Black Panther Party, and every week he's screaming about someone being racist. Last week he claimed the owners of the Houston Texans were racist. You think those owners suffered any loss of reputation? No. And really, the only time anyone is really punished for their racist behavior is when they have a superior over them who fires them to protect the company image. If there's no superior, it really doesn't matter. Look at Donald Trump. He says racist shit regularly, yet he's the current frontrunner for the nomination of the Republican Party.