Well you wouldn't understand, your not an ENFP lol. But seriously, I think this is more of a problem with ENFPs then INFPs (I could be wrong). My ultimate goal is for people to see me for who I am, for me as an individual, this is what I live for. If they reject what I say or don't see it the way I see it, then they aren't seeing me for who I am. Individualism tends to be very important for the XNFP, but in a very selfless way.
I almost think that our ideals are what ground us and if our ideals crumble then we have no grounding and this causes some serious insecurities.
At least this is true for me...
Very good question btw