There have been many times in my life when I've been inundated with emotions from others, but over time I've found ways to wall off those emotions. The times when they're strongest is when I'm under stress. I can't seem to keep the walls up, and everything comes at me, which in turn increases my stress all the more.
It was particularly bad my last couple of years of college, and the year after college. I have some diary entries from that period of time that almost sound like they could have come from a crazy person, but it was just my way of trying to deal with all of the emotions I was being bombarded with.
I tried explaining how I felt to my college roommate at one point, who I'm pretty sure is ESTJ, and he flat out told me that I was one of the most arrogant people he had ever met, and that I believed I was better than everyone else. Ironically, I had very similar feelings about him, however, him saying that really screwed me up, since that was exactly the opposite of what I was trying to tell him, and I trusted his opinions.
To answer your original question, I'm not sure how much of a non-spiritual answer there is to INFJs and empathy. I've read that many INFJs feel they have strong links to psychic phenomena, and although I'm a bit skeptical of that, I can certainly look back and see points in my life it could be viewed that way. The only thing I know for sure is that stress causes me to Feel stronger. Whether it's reading other people's moods and body language, or something else, I don't know.