being Prejudiced is one of the most miserable things about people.
I completely disagree. One of the greatest things about the human mind is the ability to make generalizations. Without that, we would be crippled in almost any task we take for granted.
When you see correlations between things, you combine those with your nativist biases (the prior in bayes' rule), in other words, the things you are born to be good at learning. Then you abstract from the data and make predictions accordingly. We've developed all sorts of techniques to reduce error over time as well.
Because we do this kind of generalization across the board, which is what makes our minds so powerful in the first place, it's inevitable that there are specific instances of this effect that negatively affect society. It's just a side effect of the way our minds work.
Upbringing and education is the only legitimate way to counteract the effects that people view as negative. Taking away the mechanism (which, of course, isn't what you're suggesting) would be so disastrous it shouldn't even be considered.