I think what you are seeing, Starry Knights, is the difference between how we treat those close to us and how we treat those that are just pretty good friends. I have very few people in my life that I would actually try to point out what they are doing that isn't working for them or bring up and issue that is causing friction between us. They would either have to be one of 2-3 people in my most inner circle (and even then it would be with a lot of thought and trepidation) or someone that I couldn't work around in any other way so have to deal with the issue.
With someone who is a decent friend (but not in my very inner circle), I would probably tell them what was wrong if asked, but if I didn't foresee a productive outcome to the conflict it would engender, I would just withdraw, see them less frequently or keep things more superficial.
I don't think INFJs will back down a lot if they feel something is extremely important, they really don't like bringing up problems, partially because they know how deeply it impacts them when other people do. It's a painful experience. They only do it if they are absolutely certain how the other person will react and have either judged that the person will respond positively, or that the benefits outweigh the possible costs. They often don't realize that they are creating more trouble for people by not addressing issues openly and that it makes the other person feel shut out and like the INFJ thinks they know what's best for everyone.
The INFJ also usually doesn't realize that others don't always feel criticism as personally as we do. I have noticed that ENFPs are much better at handling criticism objectively than we are, as long as they feel accepted by us. We tend to feel rejected no matter what and deeply ashamed and embarrassed (or if we don't respect the person we'll be bothered by it, but dismiss them).