You just need to communicate that when you have a critical attitude or withdraw in response to hearing something you don't like that it's not a final judgment. Right now, they take it seriously because it seems very serious. The package it comes in seems a lot more rigid and immovable than it really is.
My INFJ certainly fits this. Especially does his initial demeanor with people appear critical. He frequently is accused of being a snob, and has the dubious honor of having affectionate nicknames from his friends bestowed upon him involving phrases like "Nazi". In other words, he has a rep for being a bit of a control freak, for having very high & seemingly rigid standards, and of expecting people to accommodate his preferences.
The reality is, he's a lot more flexible and forgiving than he appears, and he's definitely eager to serve others & please them. Until people realize this, they often check in with him regarding things so as to avoid his critical eye. Frankly, it's less about respecting his opinion so much as shutting him up & keeping life pleasant.
A major related flaw is that he complains about all he does for people, and that he doesn't get the same in return (such a tired FJ complaint), but he fails to recognize and comment on things other do for him. Even if someone does not do something to your liking, you can recognize their effort, their motive, the message of care they're communicating. If you do this (and hold back on nitpicky criticism), then people are more likely to do for you. Otherwise, they will just be discouraged, thinking you're impossible to please (and rather whiny also).
What's funny is, I have yet to meet one INFJ who can take the same criticism in return. They hit the ceiling when they get that treatment in return. They want to be praised & acknowledged, etc, and they can't fathom that they get it wrong sometimes. They don't seem to realize how much people say "good job" when it was not exactly to their personal preferences, but they are mainly commenting on the effort. It may be a Fe thing to not recognize the very personal aspect of some preferences, as they can see things in terms of standards regarding values. Some effort to recognize that different ways from your own are not inferior is useful in tempering criticism.
This especially annoys me as my ISFJ mom still kills his cooking, and when I make her recipes I KNOW they are good, and I KNOW I have a discerning palette (and a much wider one). I've even been embarrassed a few times at friend's dinner parties where he won't eat something or he even corrects the hostess' cooking method. It's just RUDE... Then he wonders why people think he's a snob & hard to please.....They ask him for advice or heed his input to appease him, not because they highly regard his opinion....
What I'm wondering is how much of this stems from feeling threatened.....INFJs, do you feel threatened when someone is or is treated as much of an authority on something as you? Or in other words, when they don't need your input? The real question may be: Do you need others to need your approval?