In my case, I think it has to do with how Ni works. Often times I'll present my hunches to be more certain than they actually are. People who have personal inclinations against decisiveness tend to find it kind of alarming (how quickly I may come to conclusions), and then they become innately distrustful of me for some reason.
Other times I'll, say, use the topic of conversation as a jumping point, and end up connecting it to some other idea for the purpose of perspective, but then fail to highlight how the two things are actually connected (due to my Ni), leading people to not understand my point, and think I'm not taking the conversation very seriously. This tends to happen a lot, as my Ni can sort of drag me into some conversational tangents without any conscious effort on my part. The association my Ni makes, and appropriate analysis can just sort of fall out of my mouth, and sometimes the connection is very vague or personal, which I think people assume is me trying to turn the conversation onto myself.
I also tend to zone out/daydream a lot, and speak a lot about my own ideas, philosophies and conceptualizations, and all that sort of stuff. At times, I can also be a bit too light-hearted/ glib/ phony, socially, since at the time I'm likely focusing on something of a more internal nature.
I think this either gives people the impression that I'm somewhat thoughtless or don't care about them. It doesn't help that I find it hard to express feeling to others. That, and when people tend to verbalize their appreciation for me, I tend to just feel awkward about it, and forget to say anything in return... or if I feel pressure to reciprocate or to generate phony enthusiasm, I may inadvertently do it in a very mechanical/robotic way, which I can only see as being taken as ungenuine. I think most people have trouble knowing where they stand with me, though this is mostly due to my own insecurity with expressing emotion/weakness and all that shit. Even though I've made strides towards being more humble, ego wise, and being more expressive of my emotions... it's still pretty hard for me.
So, yeah, one of the main criticisms against me is that I exists too much "within my own little bubble" (someone else's words, not mine,) and that people don't know how valuable they are to me. I can see this in myself, and I could also see it being very annoying to other people. Sometimes I find it really hard to connect to reality, and sometimes I just can't find the motivation to do so. I could also see being annoyed with a person of this nature, and not seeing the point in trying to pry them out of their internal world.
In terms of classic INTJ criticisms, I also used to be really stubborn in arguments/debate, and would never concede to defeat. I also always thought I was right, and would enjoy pressing my perspective until the other person gave up and I "won" by default. This also really annoyed my friends during HS, but I've had my ego shattered a few times since then, and am much more humble and ready to admit errors and such now. I think these days it mostly has to do with how I neglect reality for the sake of my internal world