I do sometimes get frustrated talking to SJ's, but not so much because they don't understand me. More because of an issue of outlook rather than understanding, though. It is as if all the things they value - detail, prudence, realism - I just don't properly respect.
Generally, I think it comes down to how healthy an N is, whether or not they feel "misunderstood". I live in the real world, so I have a wide range of conversational topics. I get on great with other N's, but with SP's and SJ's I have no problem being UNDERSTOOD or understanding them. If somebody is not an idiot, they will understand plain English, so if you can't make yourself understood to them, I'd say you have a problem. I am not talking about a context where by necessity you have to explain something very complex - like a physics lecture - but about everyday conversation. There are some conversations I have with my Ni brother which no-one else understands - but then I WOULDN'T speak in the same way to my SJ cousin and then get frustrated that he doesn't understand!
If anything I would it comes down more to T vs F. If someone is very strongly F and not using Ti or Te style logic, I will find it hard to "get" their argument. ESPECIALLY Fi. Fe I will get, I will just not respect prioritizing it over Ti. Fi -based arguments, on the other hand, I will not get at all. I just can't identify.
Just my two cents.