I don't think this is unusual. A close INFJ friend who is married to an ENTP has the same complaint of her husband. I understand the wanting to say it honestly, when you really feel it. I also relate to the ENP - you CAN't make me do anything. I'll do it on my own time.
I think people are responding to how much you've shared about this lovely INFJ in your life and how happy you are with her. It smells, looks and sounds like love from over here, you know? You've also mentioned that you DO love her.
I think, sometimes, it's about what we think we're saying when we say I love you. It isn't, I imagine, the equivalent of selling your soul... It doesn't mean that we're giving up our individuality or that we don't in some way, as you said, die alone. All humans do that, don't they, regardless of whether they're single or not? It doesn't mean that we won't ever hurt the people we say it to or find patches of trouble in our relationships with them. We may even wake up one day to find that we don't love them anymore. Any of this could happen. It's life and we are human. I think of I love you the way Jennifer described it so well, partly a term of affection and partly a promise to care for the person as they deserve. The former implies just expressing aloud what you feel today and the latter that were you to wake up one day and find yourself really mad at them or not quite in love any more, you'd honor the person by staying and making it work and trying to rediscover all the little things about them that drove you to say the words when you did....that's the best we can do.
It is however, a part of making yourself vulnerable to the person which can be hard for ENTPS so concerned with losing control.
Finally, only you know why you can't say these words. Try and understand why exactly, for yourself. Think of what made you say those words before? How does your current relationship compare with the ones before where you said it? Sometimes the more invested we are, the more difficult it is to express it. Good luck!