It's really ignorance that's the problem. Minorities are going to seem threatening because the majority is often segregated from them. That way the majority can make up negative stereotypes about the minority without those false ideas ever being challenged.
I backpacked around Egypt (pre-revolution) and when I was leaving, I splashed out on a ticket on "the sleeping train" (basically a train with cabins and beds, which was a lot more expensive than the regular one) back to Cairo. I shared my cabin with a Australian woman in her 50s, who had been travelling for 2-3 weeks as part of tour group. We were talking about our experiences of the country, which were very different. Egypt can be a hard place to travel in because tourists just get bombarded (people wanting baksheesh (bribes/tips), people wanting to sell you things etc). Additionally the men can be prone to sexually harrass you if you are young woman travelling alone (Not ALL men - not even most men. But I did have a few negative experiences). But it wasn't unsafe and most people were decent, even if they were annoying to tourists. However, this Australian woman had been in one of these tour groups that are like a bubble - they allow you to experience the country without actually engaging in the culture, people, food, environment etc. She was kept separate from the people and had all these highly inaccurate ideas about them. Her tour guide had told her group that it was so dangerous to get separated from the group because all these terrible things would happen to you. She and another person did get separated accidentally and she told me this whole ridiculously panicky story about how frightening it had been. I tried to tell her that it is a pretty safe country for the most part; that I had been travelling for weeks there and had walked around by myself all the time (even at night), and it was fine. But she wouldn't listen. She went on about Egyptians and Arabs in general in a way that was so incredibly racist, inaccurate and unfair that I was disgusted. I don't even know why she went there if she felt that negatively towards them. I would be seriously insulted to discover if tourists came to my country and were that repulsed by NZers, even before they arrived there. But people will go out of their way to segregate themselves from other races and cultures, even when travelling in another country. And it's easy to keep believing that racist BS, if you never let yourself see the other side of the story, if you never allow yourself to be confronted with the truth.