1- My logic is barely ever appreciated by my principle friend, who is ExfP, and simply stares blankly at me whenever I make one of my sad attempts at a joke.
2- Being told that I am "too arrogant", "on a high horse", "self-important" and so on and so forth, when I'm simply stating things the way I see them
3- Being part of a minority. Although many people may see this as a blessing, it's more of a curse in disguise. How many times can people tell me I'm "weird", "a freak", "abnormal", or "odd" before I start to resent it?
May I add a fourth? Well, it's more of a "qft", actually.
4- What quaeresne said: "The tendency to deny/suppress/intellectualize emotions."
Those are good reasons to hate being an INTx. Glad I'm an INFJ.

But seriously, there are more people out there like you than you think. You just haven't found them yet, because they don't stand out much and are hard to find, being so introverted.
As far as I'm concerned, it's only arrogance if you're certain of something and you can't prove it.
Yes, I imagine that even though you're awkward socially, there's still a lingering desire to be liked and belong somewhere that never gets fulfilled due to... well, you being an INTx. The conflict comes from needing autonomy, yet still wanting to be liked and belong somewhere. That middle ground can be hard to find. I wish you good luck.
No one really knows how to work through their emotions without practice. Some people are better at sensing others feelings, or describing their own feelings, but no one is really any better at dealing with their feelings, or knowing which ones they should follow without making mistakes. Although I imagine it can be more painful when you don't have an outlet for the feelings, and find it hard to relate your feelings to the feelings of others.
Anyway, three reasons I would presume it sucks to an INTx:
1. Self-deprecating and perfectionistic.
2. Feels just as much other people, but can neither express it well, nor easily recognize their own emotions in others.
3. Struggling with internal and external conflicts because of the differences between their own needs, and the needs of most other people.