It's a strange thing.
I think it's only the meaning of the word that is really important (not to say that some phrases or word choices aren't inherently racist/hating). That's why I never understood why people can make any real arguments against profanity on moral grounds (without using religion). If I change the letters but the meaning is exactly the same, then nothing has really changed.
With that said, everyone doesn't think about these things and are overly zealous/righteous so a reasonable person is forced to make some concessions and regulate their word use.