Hmm, I think I initially dislike ESFPs if they are too loud or too attention-seeking. I had a roommate who was one, and she loved being praised, and it completely goes against my personality to give compliments to people who are fishing for compliments. I like harmony, though, so I started giving her compliments a lot more. It really helped. After a while, I just get used to ESFPs and am bemused by them rather than offended.
I really have a hard time with INFPs in the long-run. I'm so rational and judicious about problems. I get over things easily. It frustrates me that I can't just say to INFPs "it's okay, move on, don't take it personally." There are 4 of us INs that hang out from time to time. Two are a married couple (INFJ and INFP). One is me (INTJ), and one is the INFJ's best friend (another INFP). Anyway, the INFPs are self-hating, but they are always fighting over who gets attention from the INFJ. Very annoying. I'm opinionated as an INTJ, so I have to be very careful about what I say around that group. Every little thing can be taken as an emotional insult (which they will never frankly address). Very frustrating!
But, INFPs aren't all bad. My dad's one, and I love him! I like my friends, too. I just like straightforward rationality when I can get it.