Over time, I've realized that Fi users care passionately and deeply about a select few people in their lives and would do just about anything for them. Fe users care more generally and diffusely about a wider scope of people. That is not to say that I don't have levels of care, depending on how close someone is to me, but it's not a tap that is just turned on or off. As a Fe user, Fi's care can sometimes seem over the top or else non-existent. I'm sure that to a Fi user, Fe care seems kind of superficial and sometimes nosy.
To a Fe user, actions are really the evidence of care. Therefore a Fi user deciding they don't feel like attending a family function, even if it matters a lot to you, or them not acknowledging a person's efforts even if they feel a little invasive (or at least suggesting an alternative action that would be better) often is interpreted as a lack of care for the person or else immaturity/selfishness. Stony silence or grumpiness in the morning simply because the Fi user had to get up seems rather selfish (because those cues and attitudes convey to the group or to the Fe user that they are somehow responsible and should be doing something). Not responding when someone talks to you or acknowledging their presence in some way is taken as something deliberate, when it fact I've discovered that it is not intended in that way at all. To Fe, this is confusing, because if you follow Fe rules, you generally always know when someone is upset and then take measures to negotiate fixing it. With Fi, there appears to be no way to know and a Fe user has a much harder time just shrugging their shoulders to what feels like a personal accusation (or failing that extreme selfishness! - are you noticing a pattern here?)
Again, very little is neutral to Fe. Every action, comment, look, bit of body language is usually a cue that all is well (appreciation is being shown in some way) or that all is not well (and people pick up on it and try to adjust things so more people can feel happy with the situation). When the Fe user feels they are making these adjustments for the Fi user, but it is not being reciprocated, they can also become resentful (while the Fi user wonders what on earth is wrong, or else leaves the person alone to work it out - which is further interpreted as not caring!)
It wouldn't occur to most Fe users that a card detailing their appreciation for the Fi user and what qualities they appreciate about Fi user would be seen as an awkward present (because for the Fe user that would be about the best thing they could get!). This can even discourage the Fe user, because they are really not sure how to show appreciation (something they really need themselves) in any of the conventional ways they are used to doing. (I've found Fi people often seem put on the spot by compliments or public recognition too).