My empathy decreases when:
1. The person is being insincere (either when a person is 'performing' or when they're being blatantly dishonest
2. The person constantly complains
3. The person takes advantage of someone
4. The person makes a logically inconsistent argument
5. The person doesn't show an appropriate amount of tact and respect while arguing his/her point - unproductive abrasivity is just self-indulgent and dumb
6. The person is unable to admit to being wrong / generally unable to be self-critical
I'm not totally sure what you mean by losing empathy...if I'm actually dealing with such a person, I don't get visibly upset (usually), I just temporarily hate them. If I'm not dealing with them, I rant to someone else until I get over it.
My dad is an INFJ, and my list above probably describes him pretty accurately too. But he's more likely to let his anger show around co-workers and other people he's less familiar with, while I only show it around those close to me. And yeah, he's definitely a door-slammer. I'm explosive.