I think the main point is a conflict not of interests but of application. No doubt ENFJ endeavours are for the "good" (probably a result of the NFness) but the structured/unstructured dichotomy is problematic. Just because someone is not "anal-retentive", as you put it, it doesn't mean they are messing things up for the rest of you. Like I said, I love people planning the nitty gritty aspects of life for me: I don't enjoy it, so I'm glad there are those who do. The difficulty lies in bridging the chasm between the macrocosmic world of the INFP and the microcosmic world of the ENFJ.
For you, flights of fantasy are reserved for moments where you can "indulge" in them and for the rest of the time you concentrate of practicalities and interacting with others. For me, it's the other way round. The majority of my day is taken up in constantly wondering about things, therefore pushing out time to consider useful/practical things. I think that, sometimes, my "inability" to compartmentalise in the same way as an ENFJ results in them branding me "difficult" or "moody". You may say to yourself that, because you can do both things, so should the INFP, so why can't he just get a grip and realise that following all these details is for the best!
Also remember that INFPs are typically very perceptive. While you ENFJs are "spinning plates" and pleasing/appeasing several others, we usually "see through it" and begin to demand that you treat us differently because we've been so clever as to notice that being diplomatic to everyone is more important to you than pampering us! This is, of course, not allowed