how old are you, btw? i'm merely curious, because my INTJ is quite the touchy-feely cuddler. (<3 that tertiary Fi. )
Each type uses similar functions, in different orders. Se/Fi/Te/Ni and Ni/Te/Fi/Se. Or rather backwards order.
Using another example in which two types use the same functions in reverse order, my type: ISFP and the ENTJ. I have a difficult time with ENTJ's. I'm Fi/Se/Ni/Te and they're Te/Ni/Se/Fi. For whatever reason, according to theory and my own personal experience... I find the ENTJ reflects back my weaknesses in a negative way. I tried dating one once, and this relationship was never going to happen.
I could possibly see the same thing happening with the ESFP + INTJ combo. Perhaps with the boisterous ESFP, the traits reflect back your weaknesses in a negative way. But that doesn't mean these relationships are doomed for failure. As you develop Fi and Se, and she develops Te and Ni, you guys could probably meet in the middle. I think it just requires some work on the count of both partners. Perhaps both partners needs to turn down their own strengths. I think with these opposite pairings, it's easy to feel overwhelmed and insecure in the face of the other.