I read the original post and skipped the rest, so this is just to say that how you described yourself is exactly the same way I am, and I have had many people say they were surprised at all the warm gooey goodness inside once they got past what they saw as a fairly crusty exterior. One person said I should let more people get to know me, because I would enjoy more love, because I'm lovable once I'm comfortable. The thing she didn't realize is my values just don't point me in that direction. Life is not a popularity contest and I don't value the approval of everybody around me. I only value the approval of the people I admire -- or sometimes the people I just can't help being drawn to, even if I don't admire them. But either way, whether somebody likes me or not doesn't make much difference in my world unless I feel a kinship with them and have an instinct to want to share with them, which is not everybody. So it could be an INFJ thing, and it could be an INFJ 4w5 thing, I don't know. But yes, I have the exact same experience.
If I may give you some advice on how to handle it when people want to start involving you in activities -- thank them for the invitation and just say the truth -- something like, "I really appreciate the invitation, but the truth is, I'm a bit of a recluse, actually. So rather than say yes and then back out, I'm going to say no thank you and hope you understand." And the next time you run into that person you could even ask them how it went, just to show you have a casual friendly passing interest but no wish to be involved directly.
The only problem with that is it creates this air of mystery and makes you seem hard to get, which for some people just increases their wish to get you on their guest list ... but there are worse problems to have. :-D