From experience/observation, the SJs around me, who seem to see themselves as the experts on "normalcy" :rolli:, routinely refer to all INPs they know as strange, weird, and odd. Now, they have certainly encountered other SJs whom they found strange (and I find MANY SJs strange), but the pattern to nearly always find INPs strange is much stronger. INJs are often found strange, but because of having Je aux functions they seem to hide it better. Of the INXXs, INFJs appear the least strange (which is not saying much), IMO, because their Fe can help them adapt a bit. ISFPs can appear quite strange, but it varies more between individuals.
Looking at this from a theory standpoint, I is less common than E (or at least less desirable in society), N is less common than S and more prone to abstraction, and Pe is less structured & more haphazard than Je (although very extreme J can seem bizarre). T/F may seem subtly strange if paired with the less common gender, but otherwise I see little correlation. That also "supports" INPs as coming off the most eccentric to people, and INJs at a close second.