This is a very common problem for both INFJs and INFPs. I have had many moments of writhing with embarrassment at the very memory of a conversation I felt I mishandled or something like that which the other person wouldn't have noticed. I'm embarrassed in front of my very own self!! How crazy is that? The bright side of it is that you are very in tune with other people and what is going on under the surface. The very fact that you are worried about how other people feel makes it much less likely that you will tread on people's toes or be where you are not wanted. The problem is that it is like your internal volume is turned up too loud so it is harder to focus on what everyone else is saying and perceive it accurately. It was a big surprise to me in university to find out that many people also have that voice in their head that says, "What's the matter with you? Can't you think of anything interesting to say?" or "They're probably just inviting you along to be nice, but they don't really consider you one of them" and so on. I was shocked to discover that most people who come off as unfriendly are actually just unsure of themselves and don't hate me.
Not only that, I also realized that what made many of the kids in my high school popular was that they had a lot of people that they weren't scared to talk to, so everyone assumed they must have it together because they themselves were not as comfortable. Most of the popular group had come from the same junior high, were on the student council, dominated the sports teams and so on. They got lots of practice talking to people and most were not Ns so were more worried about what was than what might be. I wish I had it to do over again knowing what I know now. It took me till the last year of university to finally get more comfortable and as a result, many more people talked to me. I still fight with it, especially in larger groups with types that I don't share much in common, but have watched my ESTJ (ex)boyfriend win people over just because he is not scared to go over and chat. When we were together, people would always pass on invitation through him to me because he was the talker.
When we are not confident, we focus a lot on ourselves. I discovered that most people really don't care about me and what I look and act like as much as they care about THEMSELVES and how they look and act! I also found that the people that go out of their way to make others look stupid are just trying to draw attention away from their own perceived faults, hoping that if they attack first, no one else will focus on them. Once I realized that, it became much easier. For people like us, we for one need to practice. You wouldn't expect to be good at basketball after only a couple of tries at shooting hoops and conversation is the same way. I also found that other people love to be able to talk about themselves and feel that someone cares. They are already an expert on the subject, it is easy for you to think up questions on it and they have lots of material to discuss. One way of turning down your internal volume knob is to try to focus your attention outward on how you could make other people feel more comfortable. Everyone is insecure enough inside to appreciate someone making overstures to them and making feel welcome. Also think ahead in foreseeable situations about things that you might have in common with the people you will be talking to and what you could introduce. This cuts down on nervous chatter or awkward conversational gaps.