Yeah, I had forgotten that both ENFJs and INFPs have a inferior thinking functions. That would make sense then.
I think also that INFPs feel put on the spot or cohearsed by INFJ Fe just as we sometimes feel with ENFP Te. I avoid situations where I am going to have to debate something that I haven't thought sufficiently about or where someone tries to impose a solution on me when I think they don't understand the problem. I could see how INFPs would feel similarly about my Fe smothering their Fi or putting them on the spot to explain it.
I also expect that the reason Te users find the meandering and many shaded thoughts of Ti puzzling is the same reason we find Fi puzzling: we can't imagine not having it readily available to express what you feel. Te users can't understand not being able to readily express your logic without getting all quanitifying and rabbit trailey about it.