I learned from a young age to adjust my behavior to those around me in order to have a frictionless existence (family issues), which means I don't overtly butt heads with people. However, from the people I know, ESTJs undoubtedly cause the most aggravation. I don't get angry easily, but my ESTJ sister has this ability to make me pretty wild (Very rarely to her face. More likely a massive rant in my head, or out loud to my INTP... who seems to find it vastly amusing.)
Another ESTJ (INTP's cousin), he can annoy me, but not in a fury like my sister. Mostly he's just irritating in an amusing way. And he thinks I'm weird, so it's entertaining to perpetuate the idea. My INTP has trouble getting along with him... I've known other (suspected) ESTJs only as acquaintances. Even when I can
see that they're nice people, and I am superficially fine interacting with them, there is something about the Te that rubs me wrong. I kind of find myself shutting down.
I don't know what exactly it is that bothers me... this firmness of opinion? A slightly (or not) in-your-face quality? They often (thought not the two ESTJ family members I mentioned) shoot me down without giving any apparent thought to what I've said, and generally their rebuttal is based on a false assumption, but I don't even want to bother with trying to explain when they're not going to listen.
I'm not interested in trying to change people's opinions, but that's all they seem to want to do with me.
Sometimes it feels like they are bulldozers and I am standing in their way. I don't want to puff myself up all big and loud so they stop, when the whole conversation is pointless to me anyway. I'd rather transmorgify to a little woodchuck and burrow quietly away.
Oddly, the ENTJs I've known don't really bother me. I mean, they
can, but they have qualities that generally ameliorate the annoyance.
The ISTJs I know can be mildly annoying, but hardly worth mentioning. I just don't have much desire to talk with them. Conversation is a struggle, it's a stretch to find things in common to talk about.
And the INTJs I know are great.
Oh and as a side note, ESFJs annoy the holy hell out of my INTP.
(Disclaimer: I'm only speaking of people I have known, and my personal perception of them. This is not meant as a blanket statement for a type.)