In my experience, the worst part about being an INTJ seems to be the fact that most people don't understand me and are often offended by me or the things I say (such as, "no, that's not a good idea and we're not going to do it." or "I'm sorry, but that's just not the case. You have been misinformed." or "No, you're wrong.")
Also being an un-girly girl who is independent, thinking, outspoken and sarcastic doesn't make you a very popular girl. Which has always surprised me, as the typical female archetype has always kind of pissed me off.