Well, think about the words self and image and you'll probably get your answer.
Nothing is more important to a four than having an intimate knowledge of the self. Ideally, that self is what a four would like to be known for, but fours also withdraw to protect their feelings so the 'real you' is often hidden. So with that comes a lot of frustration. You want to be known for who you really are but you're scared of showing what you perceive as the real you. And with this comes the four image. Fours can appear deep and mysterious because much is hidden. Enigmatic, too, because something will be shown and then retracted, so then who knows what is what. Often fours like to share the 'real them' in indirect ways, like through art. That way they think they can have it both ways. Stay withdrawn to protect their feelings, but express who they really are at the same time.
I can't speak for all fours, but I know I like it when I am referred to as deep, mysterious, enigmatic, complex, creative, authentic. So if someone says I'm something that counteracts that, it can be troubling. Say someone were to refer to me as "The Girl Next Door" or "A Nice Person," and not mention anything about my complexity, etc, it can freak me out. What are people seeing, I'll wonder. Obviously I'm not doing a good job of presenting the 'real me' correctly. In other words, I don't think they are right and that I'm wrong so much as I'm not doing my job in presenting myself in the right way. It's entirely possible that I do possess these qualities that they mention, but they do feel threatening to my ego, I guess.
A healthy four would not give a crap about any of this. They would just do what feels natural and not worry about presenting the 'authentic' image, they would just be authentic. As a four, this image-consciousness is incredibly difficult to shake. The irony, of course, is that the art I create that I feel is most true, that I am most proud of, is when I lose the self image consciousness and just do.