The problem is when Fe says there is ONE way to act, when it sticks doggedly to an approach, regardless of other approaches being just as effective. It feels like a willful misunderstanding to an INFP; the person refuses to acknowledge another possibility as just as good. I realize this might seem more SFJ, but occasionally an NFJ fails to see another angle or more often misses the details that eliminated what they see as a better option.
As neither an SFJ or and NFJ, I can only speak for myself, which may be a different perspective than what you're thinking of, but I still operate very much according to Fe. Here's how I would handle a situation in which we're all trying to come to a compromise. I would listen to all perspectives, trying to figure out which would suit the needs of the most people, or failing that, the intended purposes of the group. I would realize that this means I myself might not get exactly what I need from the compromise, but that it might be better for the group overall. I will make a suggestion as to the best solution, and if that goes over well, I'll work on making everyone happy and refining the solution, possibly including a plan to accommodate other members more fully the next time. If someone pipes up with a better alternative solution, I'm absolutely open to that. Fe, for me, is not about excluding good alternative ideas. It IS, however, about making the majority of people happy, or at least being fair.
As Hopelandic brings out, an INFP is expressing via Ne, which means MANY possible actions & outcomes are seen, and the one which seems most appropriate to the Feeling is often chosen. One downside, is the INFP often chooses the most ideal, as opposed to the most realistic. Another downside is, the INFP is not speaking Fe language, which results in misinterpretation. FeSi doesn't know what to do with any approach that is not familiar, and FeNi seems paranoid in its interpretation at times, ready to assume the worst, and doesn't even let the action play out before jumping in and condemning it, inevitably making it bad. The readiness to form judgments too quickly is a real problem.
I have no problems with Ne whatsoever. I like creative suggestions. If it's mostly about being heard and exploring possibility, I'm definitely cool with that. I'm a P, though. I do think you are right about the Fe/Si thing and staying with what's familiar--I have seen that many times. I have an ENFJ friend who is VERY opinionated and decisive. You can suggest something to her, and sometimes she's fine with it and open to trying something new. But sometimes, she'll clamp down on that idea with a NO so fast you want to slap her. She's given it no thought, but she knows it won't work, and we're not doing it. To her credit, though, if she respects you, she will usually go back later when she's alone and think about it, and then come back with a new perspective, sometimes having completely reversed the earlier decision. I can't be so J about things unless I have already Ne'd the possibilities to death. Sometimes I have to shut down suggestions made by students because, though it's my nature to be flexible and I wouldn't mind accommodating 2 or 3 people, accommodating the individual needs of 60 or 80 students would mean that I would be the one stuck doing all the extra work. I've already foreseen the outcome of those suggestions, or in some cases tried them, and it doesn't work. So I have to be less accommodating in those cases than is my nature. They will have to accept not having their ideal circumstances, but I've done my best to be as flexible as I can without burying myself in the process. It's a balancing act.
The Fe person also seems to think that their ONE way will affect the INFP as it does everyone else, but here again, there is something lost in translation, where the INFP just sees some seemingly phony one-size-fits-all approach that is devoid of deeper meaning, and it's almost insulting to our intelligence. Then the Fe-er gets mad that the INFP is not affected as everyone else is affected, and the INFP must not be "normal" to not feel & respond as everyone else does. In reality, the Fe-er has failed to really consider the effect of their actions in relation to the individual. At best, Ne allows an INFP to accommodate individual needs very well, to adjust the approach as they go. I think tertiary Si comes in handy here, as the trial-&-error is "documented" mentally and the INFP has patterns & experience to draw on instead of groping in the dark for the best approach. No doubt the Fe approach works well, and it's refined to suit as many as possible, but it seems there is far less adjusting for the individual. I do imagine Si & Ni aid in this for them, as well as in interpreting new approaches foreign to Fe protocol. Obviously, Fe people do adjust to individuals at times also, but it seems to take more time with the person, and they have to be more aware of not allowing first impressions to hold more weight than they should.
I think this strikes at the core of Fe/Fi interrelation, from what I understand of it.
As an INFP, I sometimes feel repelled by Fe dominance, although I am aware that, most of the time, that was not Fe's intended outcome. As OrangeAppled notes, the problem seems to lie in the construction of plans/outcomes by Fe which only seem to have taken account of my feelings or thoughts in a very cursory way. The problem occurs due to what I perceive to be the artificiality of it all and the reductive nature of Fe attempts to identity and control all the variables, which is what the INFP necessarily becomes. It is as though, through Fe attempts to accommodate everyone, they have not realised that perhaps the INFP wanted something different. Voicing one's concerns about this to Fe can sometimes result in Fe feeling offended, perhaps because s/he feels hurt that the INFP is "revolting" against their best intentions. Similarly, the INFP thinks "well you've accommodated for everyone else, but you don't care about me! Therefore, the others are clearly more important to you!". Fi feels overlooked and Fe fears some kind of maudlin insurrection.
This is meant as a serious question, because it strikes at the heart of the breakdown in Fe/Fi interaction in many cases. Many Fe users see the above reaction as sort of a "special snowflake syndrome." Is it genuinely difficult to accept not getting your ideal desired outcome? Does compromise of any sort lead you to feel this way? This is what Fe doesn't understand. We're generally fine with not getting our ultimate desired outcome if the compromise is fair. To us, not being willing to accept any perspective but your own seems, for lack of a better word, immature. I know that's not where you're coming from, but that's what it seems like to an Fe user, so I realize there's a disconnect thing happening here. What would you consider to be a good outcome with a Fe user? One where you felt valued. Are you okay with a compromise outcome if the Fe user heard you out and explored the possibilities, but ultimately felt that it wouldn't best suit the needs of the group?
I think what's hard for me to understand is that I can't imagine there ever being a situation in which each and every member of the group felt like every shade and nuance of their opinion was validated and a solution emerged in which everyone was 100% happy. Making one person 100% happy is inevitably going to make the next person 50% happy or 2% happy or 0% happy. (Also, "happy" looks weird when you type it over and over.)
Also, regarding the not caring about you, but caring about the other person, I'd say that is usually not the case. We just haven't cared about you MORE than the other person.