I think we have two different views on type, you seem to be using MBTI to type people. Type in Jung's view is the precondition for a person to think a certain way.
There are alot of other important variables to a person's behaviour, liking upbringing, the current environment, and other sociological conditions. Type only plays one part in the complexity of human behaviour. The way people behave is a dynamic process of different influences and conditions.
It is important in Jung's idea of typology to separate a person's outward behaviour from their inner cognitive processes. Most of the time a person's outward behaviour will conicide with their cognitive disposition, however, you can't observe a person's outward behaviour and then say you know everything about that person, or know what that person is really thinking. People will not always act in the way their type entails them to.
And you say the annoying quality is intrinsic to the INFJ type, please explain this connection? Just because you meet a few INFJs that you deemed annoying, how does that make it intrinsic to all INFJs?