Ni is so interior in a lot of ways that it would surprise me if it had a polarizing effect, but I think your assessment above about the INFJ could be correct - they react to him the way he reacts to them.
In my case I think the Ni manifests most strongly in the sort of intellectual/artistic connections that I make, mostly inwardly. It also manifests in my internal struggles when I'm trying to figure out a complex situation in my mind, or why I've reacted to something a certain way, etc. I relate to people very much through Fe, and I do tend to do the diplomat/placating/let's all just get along sort of thing. So it's unlikely that I would polarize people.
I think that a lot of Ni-doms use their Ni to make strong judgments about others and what they are like, sometimes to the point of saying "my intuition is never wrong about what others are like" and that sort of thing. That sort of Ni use would be more likely to polarize, I think.
When it's your friends and acquaintances who have a polarizing effect, rather than you yourself, sometimes it's a bit hard to know what's going on. A lot has to do with how the individual perceives things, of course. One of my friends, who is a likely ENFP, reacts very very intensely to perceived slights and will tell me that the friend in question totally let her down, rejected her, stabbed her in the back, or whatever. But in many cases (though not all) she forgives quite quickly and they're friends again. So it makes me think that "she rejected me" and these sorts of comments should be taken with a grain of salt. Or she'll say "all those mutual acquaintances of ours HATED ME"...but I seriously doubt that was the case, from anything I ever saw or heard. I think most of them were just indifferent to her, but she took that as open hatred and rejection, which I wouldn't do.
Then again, sometimes others have gossiped directly to me about these people, or said "I used to be friends with her but just couldn't any more" or whatever, so obviously the polarizing thing isn't all just in the individuals' minds. It's just really hard to know sometimes.