I will preface by saying I am drawn to certain INFPs for romantic relationship and friendship. Or people whom I perceive to have a lot of Fi. And even though I personally am really drawn to them or value our friendship, I know these specific INFPs are not everyone's cup of tea.
I think aside from the 'outlier' personality thing -i.e. not stepping in line with popular convention and society (which can be said is an issue with ENXPs) is that INFPs like INTPs are generally ill-suited to navigate social waters and communicate their decisions/behavior to the outer world clearly.
Being Fi dom myself, I know how 'mercurial' and 'quirky' my behavior - which is caused by my internal thinking/feeling/processing centers - can seem. However, being an extravert and generally social I can compensate better than an INFP can for my inner quirkiness by adjusting my outside behavior to be more consistent and reliable and *understandable*. I also think I can predict and 'see' more clearly what other people are expecting and what social repercussions my actions will have and head off potential trouble by vocalizing. I think INFPs not as much.
And INFP by choice and nature don't generally make the effort to explain to the world why they do what they do. On a personalized level, this just means that sometimes an INFP will be hot and cold or 'dislike' you for no reason. Sometimes they actually don't like you, sometimes it is just general aloofness.
It's easy to misread an INFP which I think happens often, but I think often the INFP doesn't make an effort to be understood or see how their behavior is affecting other people. There is a definite sense of being somewhat apart or in their own heads. So I don't think this means people 'dislike' INFPs, just that there isn't much sense or effort made by others after a while to try to connect with an 'uknowable/unpredictable element'.
Basically, INFPs can seem like islands, oblivious or uncaring about the worlds around them and are poor commnicators. I have seen INFPs be asked point blank questions and 'lie' by saying something is fine or they 'don't care' about something. When they obviously do. They can blow off other people's efforts to try to reach them and talk to them.
You have to be very close or very trusted by an INFP to be told exactly what is going on inside them. Since most people aren't privy to this information, to other people an INFPs behavior can seem all kinds of puzzling and put further distance between them.
Sooo, I don't think this means INFPs are 'disliked' just not as socially connect in general. And definitely often misunderstood.
In general 'prickliness' and 'aloofness' and 'out of it-ness' are not welcomed with hugs and parties. I think INTPs also tend to suffer the same fates for a lot of the above reasons.
I don't think it's anything to worry about as most of the time being introverts and Fi led, I think INFPs are quite fine with the way they operate and don't have a desire to be known widely. However, I have definitely seen cases of unecessary friction and misunderstanding happen with some INFP lovers and friends where they honestly don't seem to see the part they played in it.
Then again, I've had pretty tumultuous dating situations with INFPs and my suspected INFP friends have strong personalities so perhaps all the above doesn't apply in general to other INFPs.