I haven't known many INTJ's, but I work closely with one, and though I personally have no problem with him I think that's only because I'm a brash little SOB and tend to pay little heed to things other people consider warnings/alerts, and tend to bowl in anyway trusting in my blagging skills to negotiate any trouble. And that's what I have to do with my INTJ quite often.
So here I'm only talking about him, but obviously since he is very INTJ, there may be things that carry across to others of his type. IOW I'm not generalizing!!!
He can be very frosty - even when he doesn't mean to be. Because he's quite minimal with his words and even more so with his emotions, he doesn't give anyone any reason to think that he likes them or respects them. True, he doesn't give them concrete reason to believe he doesn't, but people generally tend to be a little insecure and so if in doubt, will assume the worst (that's where I'm different). If somebody comes to talk to him, to say good morning, just trying to be friendly, he tends to sorta look at them while they talk and not say anything, with a pretty inscrutable face, so for all they know he could be thinking "Will this insufferable moron ever shut up?" - he probably isn't (though sometimes he probably is!), but that's how people take it. He gives a minimal reply most of the time and seems to want to get back to work, so people feel rebuffed/dismissed, and because rejection hurts, people don't tend to try again.
I'm just a pain in the ass and I keep needling him until I get a rise (like Q with Picard!), but most other people are kinda scared of him, not finding tempers as amusing as I do. I can usually make him laugh but it's often despite himself.
Also, he's quite inflexible in his views. He believes he's always right and knows everything; that there's nothing of interest that anyone else can tell him. That's a slight exaggeration, but not much of one. And the way he behaves and speaks to other people gives the impression of that attitude being much stronger than it is. Sometimes somebody says something to him and he makes a sorta surprised look, as if to say "I didn't think you had it in you", which can be quite insulting, though he means the opposite. IOW he's telling them he thinks they're a moron and is surprised to hear them say something intelligent.
Quite often someone might approach him with some P type Perceptions, pointing things out he's missed, but he doesn't tend to listen a lot of the time unless you're willing to really push it through his anger barrier (which I do) - most people aren't as comfortable with argument/anger as sport as I am, so they leave him in his errors for a quiet life.
Basically to the average person, he makes them feel like they're just in the way, of little value or interest to him, and gives little for people to respond to. So that's why people avoid him - they don't like being made to feel worthless and stupid.
That said, he has many fine and lovable qualities too, which is why I keep him on and enjoy his company. But you wanted to know the things that put people off about INTJ's, so I focused on those sides of him.