That's also why I keep going to type forums asking people what they think my type is; I can't figure it out myself because of all of the subjective factors and I'm trying to get information from others (while simultaneously doubting it).
To take yet another shot at my Fe usage (I've been observing myself today) I've come to the conclusion that I see people as an annoyance, but still care as to how they feel. Paradox? I don't think so.
What I realized is that the essential thing that I crave from other people is respect. I want to be admired, I want to be told by other people how awesome I am, and I want to accomplish things that other people can see and enjoy. I often beat myself up when I compare myself with other people because they seem to have visible accomplishments while I don't; thus I feel like I don't deserve attention or respect, something that I desperately crave. Thus there's this awkward love hate relationship with people; I like people to pay attention to me but after an extended period of time I find myself annoyed, spiteful, and jealous. It's nothing that I'm proud of, but that's how it is
What's more is that most of my fear from other people comes from me not wanting other people to think ill of me. I want them to look up to me and like me...and I think that I'm (willingly?) misinterpreting that as genuine empathy. This often manifests as me wanting people around, but having an inner lack of concern with them or their personal concerns as long as they aren't thinking ill of me or hurting my feelings. I'm not very proud of this, but in the past I've found it very easy to lie to people and play two sides of a conflict. Fe creatives probably would explode from doing something like that. The only time when I'll actually feel bad for lying to someone or the people that I actually care the most about are my closest contacts; parents, sister, close family members, long time friends...and even then the difference in warmth and empathy is a small one.
I guess what makes it hard for me to type myself is that my psyche seems to have worked extremely hard to obscure the main aspects of my personality....probably out of guilt.