Yeah, I've experienced this type of thing many times. In this case my imagination has always worked against me. I imagine that people hate me or that they are in love with me, when in reality, they're probably not even thinking much about me at all. I never seem to relate to people normally in my head, and always have to "come back down to earth" and remind myself of what's actually going on, because if I don't, I can accidentally offend or surprise people with my arrogance or jumping to conclusions, etc.
I think it's also because I am always thinking in metaphors and symbols, and the slightest things that mean nothing to others mean things to me, so a lot of times I pick up cues from others that they probably didn't even intend to give.
Actually, it's something I hate about myself. I never feel balanced in that way, and I feel too self-absorbed. It's like there's always some sort of crazy story going on in my head and I have to work so hard to make my brain correspond with reality. :/ Actually, I feel embarrassed admitting this right now. The only way I console myself about my wild and ridiculous imagination is by planning to write a really good novel one day. Haha.
One thing that works against us INFPs is thinking that everyone feels as strongly about things as we do.