I have to as people are constantly changing. My friend might not be in the same mood as he was yesterday, or even an hour or moment before this one. If I am to interpret how they are right now, there's only so much use I can get from considering how they were last week or a year ago. I am concerned about their needs and thoughts right at this moment, because that has the biggest affect on our interaction right at this moment.
The essense of who someone is to me doesn't change much, only when they evolve and mature, making major changes in their thinking or emotions. And even then, they aren't a different person to me, I just have a new, deeper understanding of who they were all along. Just like how they appear to themselves - at the very core, they
did not change, only their understanding of themselves and life changed and they are now thinking/feeling from a different perspective.
It's important for me to see people as ever-changing, because that's how I am myself. If I want them to accept me as I am in this moment, and judge based on who I was a year ago, I must do the same for them.
Perhaps for you it is the opposite way? Do you remember so many details about people, you are more able to address their needs via Fe because you have a more long-term memory of what they like and what they need.