I was mulling this over, and I think the previous posters have definitely hit on various factors that play into the phenomena: INFPs being slow to set limits when young, not demanding reciprocity, not appearing confident, etc. I know when I was in my early to mid 20s, I tended to attract those with borderline personality disorder.
Still, I think there's another factor that involves the way young INFPs can feel isolated and look for a kind of resonate introspective emotional depth in other young people. While this can help one find other IxFxs, it can also select for those who have turned inwards towards emotional introspection because of trauma. Therefore, we may naturally gravitate towards the "broken" when young, because they may be more emotional aware than the healthy of some types.
Other types, when young, may only turn towards emotional introspection when in deep distress and forced to do so... I suspect that's one reason Fi gets such a bad rap—if the only time one is focused deeply on one's emotional state is when one is deep distress, of course it's going to be unpleasant.)
Plus, INFPs tend to cheer for the underdog, all else being equal. That quality also leaves us open to people on the fringes. While not pre-judging those who don't follow societal expectations can be a plus, it may also leave us more open to those who are emotional unhealthy.