I have no idea. I've tried to figure out my own difficulties with such things, too. I eventually rested on the fact that I am an insecure people-pleaser afraid of disapproval. Or figurative gaping holes in my psyche where Fi and Te are supposed to be. Or something. I'm certainly not claiming this as the case for you.
I do think it's interesting that we are opposite types with this same difficulty, though. I wonder what that means, if anything.