I don't think I've ever met an INFJ, let alone been involved with one.
The closest comparison I have is an ambiverted ENFJ. We were going through similar struggles so our bond evolved naturally through supporting each other. Communication issues and problems with mutual acquaintances kept it from working out.
The initial pull was noticing that they always kept something of themselves back even when they were actively participating and I wanted to know what and why. She was also witty and disturbingly skilled at making sense of my jumbled up thoughts and rambles.
The crux of the communication issue was that at the time she was an appeaser and I was excessively bullheaded. Our approaches to solving problems were too different and eventually I started to feel fluffed and became more critical of the things she said. Cue resentment and distancing loop.
I could try to oversimplify things and distill it down to Ti/Fe miscommunication but it would be a cop out since the real excuse is that one unhealthy party drastically reduces the chances of a relationship working out. Two unhealthy parties and you start to enter the realm of the inhumane.