The only types I've had consistant annoyance with are ExFJ's.
ESFJ's and I seem to have very little in common. Even where I become friendly with nearly everyone in a work environment, the ESFJ's seem to always be beyond my reach. I don't want to approach them for some reason and the feeling seems to be mutual. They give off a stuck up vibe or maybe a 'fakeness' that I can't stand.
ENFJ's and I have mixed interactions. Its either really good, or REALLY bad. The main issue with them seems to be communication. There are ENFJ's who completely GET me, and will say what I'm thinking almost before I think it. They're compassionate and warm and are the ones I call when I need advise. Then there are the OTHER ENFJ's, who I feel I'm speaking a different language to. Miscommunications are frequent and very severe. They'll get on my last nerve within minutes of being around them and I just want to wring their necks! Eventually I just end up avoiding these ones because the interactions with them drain so much out of me.