What they see is a person who preemptively bypasses interaction in the present moment in favor of listening to a long-term vision that does not take into account a number of factors that other people deem important. Let's contrast it with Ne a bit to explain. Ne is more in favor of remaining closer to this actual moment. What do you and I make together in this moment? Are we enjoying each other's company, is this beneficial for both of us today? If yes, Ne will build a thousand tomorrows on today's assessment. When Ni says, "I won't be friends with this person because 5 years from now we will part ways" you miss 5 years of caring friendship even if this ends up being true, which perhaps wouldn't be a big deal if life spans were infinite, but they're not.But still, I'd say if they know me at all, they should at least know if I'm a good or a bad person and should be able to answer the question does their theory of my evilness suite what they know of me.
If you lived to be 100, you just let 5 years of "today" go. You let go of potentially 1825 days of friendship. 5% of your lifespan.
It goes the same with "getting rid" of your dear ESFP friend. You let go of thousands of days of caring friendship because you're both human and make mistakes, because he called you a dictator? (I have bad news for you btw - to an ESFP, any J type can seem very oppressive, try putting yourself in those shoes, think of how much easy-going adapting he's probably been offering to you over the years). And now you face the prospect of thousands of tomorrows without that friendship and the years of work already done to nurture it? You're not happier. Now, I'm not saying all combinations are good matches or that people don't grow apart or past each other, they can. I would simply re-examine the pros and cons and perhaps make a working path to at least healing what was in favor of what could again be.
You're not Ne, nor should you be. But there are lessons from your extroverted cousins to consider, just as they can learn from you. You seem like a nice person and I wish you every happiness. Letting go of today's interactions though are the key to being alone today. If you don't want to be alone today, interact instead.