Exactly. If you're watching an INFP for a while, they'll definitely do some things that seem Fi and some that seem Fe. (More that seem Fi, I would think )
The trap that I see people falling into, though, is confirmation bias-ing themselves into labeling all uses of Feeling from that person as Fi (because you can always come up with a narrative that could explain how it could be Fi). The problem is, you could probably describe a lot of those behaviors as Fe just as easily. People tend not to call the behaviors Fe, though, because they want MBTI to be all nice and neat. That's the problem I'm trying to articulate here.