Anecdotally, it would be interesting to hear people's experiences.
I usually get along okay with ESFPs, because they're SO easy-going and fun. And I'm flexy. They might be Se, I might be Ne, but we still usually play nice together.
Usually where things sour: I think they are being irresponsible or illogical and not following a path that meshes with their goals when it makes sense (they definitely don't easily connect cause with effect), while meanwhile they wish I would lighten up and not be so bound by thinking through everything and instead just need to leap into things more often. There's usually not a lot of depth for me either; they tend to focus on the positive and fun experiences (although they are willing to get all philosophical as part of their flitting around), I usually want something that goes a lot deeper and takes more focus.
ESFJ and INTP share the same functions, they're just in the reverse order... so how much does order impact connection?
We could also think about "faces" for a moment, or function pairs too.
INTP = Ti + Ne AND then Si + Fe (= ISFJ)
ESFJ = Fe + Si AND then Ne + Ti (= ENTP)
In some ways, I think INTP and ISFJ connect a little better, and ESFJ and ENTP should connect a little better. Because the functions pair up better.
Let's then look at inner face and outer face:
INTP = Ti + Si (inner) and then Ne + Fe (outer)
ESFJ = Fe + Ne (outer) and then Si + Ti (inner)
ISFJ = Si + Ti (inner) and then Fe + Ne (outer)
ESFJ = Ne + Fe (outer) and then Ti + Si (inner)
Here again, note that even if the faces are all shared, the INTP and ISFJ rely on the inner face first usually (because they are introverted) and ENTP and ESFJ rely on the external first.
Again, this is all theory. Hearing practical experience will help inform theory.