My first girlfriend was an INFJ who was also a HSP. I was 16 she was 22 and were together three years. We were each others first love and lost our virginity together. The love buzz lasted about a year and a half. Without getting into the details of the nature of the environmental conditioning, parental schema or societal factors each of us brought to the relationship table I can sum up the later portion of our relationship in one word tumultuous. Ok two words, immature

.
Imagine two guys going up into the air for a high five and completely missing. This happened frequently and was easy to get over when we were "in love". In the end this is what it was like from my perspective. Nothing I could say or not say would be taken well. Nothing I did or didn't do could go over well. I was constantly under a microscope and walking on eggshells. I was constantly being judged and under those conflict conditions I retreated into myself and that too was judged overtly.
I was initially the person who always wanted to talk things out and compromise and focus our energies on positive things. After the love buzz wore off I had difficulty reestablishing that emotional connection we once had. She simply could not explain her feelings well unless it involved a judgment and a course of action. The intimacy was gone from her. I felt like there was nothing to grab onto emotionally to stabilize the relationship or myself nor to even help her. I need that connection, it strengthens me, it motivates me to continue in the face of adversity, it is the sanctuary without which I feel lost.
This is a quote from a website about the nature the push/pull dynamic.
Expectations and Hidden Issues
"Look for the SWAT signs.
Scorekeeping - when one or both of you are keeping track of who does what.
Wheel Spinning - when you talk about the same problem over and over again. When an argument starts with you thinking, "Here we go again."
Avoidance - when one or both of you are avoiding certain topics or levels of intimacy.
Trivial Triggers - trivial issues are blown up out of all proportion. A small event triggers horrendous arguments.
Many hidden issues arise from deeply held expectations.
Expectations build up over a lifetime of experiences. These expectations are based in the past but operate in the present. There are three primary sources for our expectations:
1. Our family of origin,
2. Our previous relationships, and
3. The culture we live in.
Expectations are transmitted both directly by what we hear and indirectly by what we observe. A hidden issue can't get triggered in the first place unless an expectation is violated. Studies show that it's more likely that relationships will develop problems when expectations are unreasonable. Conflicts caused by unexpressed expectations are very common and unmet expectations can lead to great disappointment and frustration in your relationship. One great clue to expectations is disappointment. It's a good habit to stop a minute when you're disappointed and ask yourself what you expected. Doing this can help you to become aware of the expectations that may be unconsciously affecting your relationship.
Be reasonable in what you expect. Be clear about what you expect. Unless you make your expectations clear, you'll have trouble working as a team. You can't work from any kind of shared perspective if you don't share your perspective."
Couple her strong push/pull tendencies with her HSP personality and a sensitive INFP and you can imagine the conflict. Granted, I was young and had not developed an emotional shield for specific use in a romantic relationship. Additionally my perceptions were flawed as to the nature of her judgements.
Advise for an INFJ:
1) “It is not what you say that matters but the manner in which you say it; there lies the secret of the ages.” ~ William Carlos Williams
You can be as judgmental as you want with me, if you word it right I will not only notice this (Fi) and think its endearing you are doing so but it is highly effective at 'swaying" me. I also need time to digest this information. There is nothing I abhor more than someone who is not only judgmental and argumentative but expectant that I change my mind on the spot simply because 1 person came along and felt the need to point things out and then feel the need to change me. This to me is the nature of the conflict between the types. How the INFJ expresses their judgments and expectations and how the INFP perceives it and deals with it.
2) “I've learned that people will forget what you said, people will forget what you did, but people will never forget how you made them feel.” ~ Maya Angelou
This is true for most people, but more so to these two types. Instead of focusing on the rightness/wrongness of this quote assume it is very much real in an INFP. This is very addressable however, prod them to talk about it after a short time has passed, and then whatever you do DON'T judge them for having done so. (really long article on emotional validation
Validation) An INFP can and will forgive almost anything if given enough time and validation.
I wanted to post more, including an advise part for INFP's but real life barges in on my thought process and I've lost the flow. Perhaps later on.