The more INFPs may appear to have their "walls up", the more they are paying attention to getting to know you.
We are intense, articulate inner-feelers, so if we decided to show our true selves to the rest of the world at first glance, the response we would get would make us "feel" isolated and misunderstood, even more so than the INTJ or the INTP.
Why? Because most of the world doesn't understand the true essense of inner-emotion, and usually mistake it for the absense of civilized, orderly behavior (especially in western culture).
It is evident that most INFPs have a difficult childhood, and one should note that we actually ARE quite open as children, and usually very confident to begin with. However, we naturally get shut down and misjudged by others, and because we have a Feeling tendency over Thinking, we also naturally take this quite harshly as children.
Being Perceivers and not Judgers, we take longer to come to healthy conclusions, because of a desire to remain open-minded at all times. You can imagine the struggle of a child who has to deal with being isolated, emotional, and yet unable to be easily judgemental, therefor throwing his self-image into chaos.
The INFP tends to be amazingly gifted (just take a look at most of the famous writers of the world, as well as talented musicians such as John Lennon and John Mayer; both of which are INFPs). However...
It takes us a while to understand our place in society, as well as find balance in how much of our inner-feeling we should expose to acquaintences. I suppose this is the great sacrifice in having natural-born talent; we take much longer to catch up to the rest of society on a sociable level.
Bottom line, our inner-feelings make us both powerful and vulnerable, so we must guard this "gift" accordingly. Everything relates to balance, so it would not be fair for gifted people to have it the same as the rest.