Because there's such a thing as differentiation between actual racism and jokes. And because the people who are unable or unwilling to do that should be admonished for it -- particularly because they're the ones who insist on keeping society locked in a state where simple words can affect someone, rather than letting people evolve and immunize themselves to those words. Furthermore, treating any offhanded remark -- even ones in jest -- as racist is an affront to the principles of meritocracy our society is founded upon: by insisting that people get special consideration because of their race while ignoring the fact that doing so disallows them to succeed based on their merits as people. You enable those words to be hurtful by making them taboo, rather than simply treating them as what they are: mere words.

In short, because you're the one holding back social progress. You deserve to be admonished for it.