I think I'd originally written a post saying almost exactly what you were saying, but decided not to post for one reason or another. But yes, that is it, exactly. We need TIME, huge amounts of time to process things. If the other party is not patient enough, they will interpret this time as us being passive-aggressive. Nope, we don't even think about trying to punish you - our internal state is far, far more interesting to think about! You don't even come into it! I actually believe passive-aggression is Fe being misused. Fi doesn't bother with the other.
The time frame is important. Fi-Ne is slow, glacially slow, for the things that really matter to it. It wants to be correct. A similar thing happens for Ti-Ne - it is fast for things that don't matter (hence wisecracks etc), but if tasked with something more important, then Ti-Ne will spend lots of time cogitating until the correct answer is achieved.
DD I am very impressed! Especially with your recognition of healthy vs unhealthy members of any type, or mature vs immature members of any type. I have been irritated in the past by negative stereotypes that people are using - and they are based on having met one or two clearly unhealthy examples of what they have guessed is a particular type. I guess that's Ne talking - asking why people aren't more open to the possibility that they are wrong in their assumptions?
I'm enjoying reading about INFJs as well - I sometimes go into directive mode with a friend who's having problems because Ne wants more information, but I do it very gently, and use it with huge amounts of patience - I let (what I perceive as) their Fi have its say first. I will also ask other sources instead of the person involved, simply to get a different perspective.