My impression is that people tend to have a lot of social fears because of past hurts. There is a deep drive to avoid similar hurts in the future, and so by categorizing the people who caused hurt, we think we can avoid similar hurts in the future by avoiding others in the same category. That approach creates a terrible mess in comprehension and communication and absolutely builds walls. The human mind is so skilled at confirmation bias, that we think our points are proven, that our prejudices are reasoned. I don't know if people can get past that because it tends to be the modus operandi of most.
It's easy to forget to do, but I think that you can train yourself to
intentionally try to prove the opposite (or if not a binary, then prove something coming from another angle), which is something that you have to decide to do before you start (or even during) information gathering.
The big thing I notice with typology of any kind, not just MBTI but forming
any group identity is in-group and out-group biases. There is a natural tendency to see people in your group in a more favorable light and people outside of it in a less favorable one, which can readily become a dangerous thing (but also fosters a sort of group cohesion).
Also, I think what you're saying about social fears makes sense, but it begs the question: what if the category of "avoidables" is an accurate one? Often people make categories that don't match up with reality. A mean person was wearing a red dress, so you avoid people wearing red dresses - doesn't make sense. And then confirmation bias kicks in and people look for examples of mean people who wear red dresses and they find them. And you'd have to be more specific about them being "mean". Maybe you didn't realize you were doing something to them which elicited that response. Then again, it seems pretty reasonable that if you identify that some people aren't just "mean" (they are really just different than you, more direct) but are "abusive", it makes sense that you try and avoid these people. Not psychologically avoid that they exist (denial), but proximity/life avoid.