Warning: LONG post ahead....
My pre-school years were spent at a pretty good school in New York City. The kids were tough but surprisingly easy to get along with as long as you treated them with respect. Eventually, my last year there, I became a leader. The kids listened to me and followed me (mainly because I was the oldest, at 5 years old). In any case, I basically fit in there for some reason. It was probably a very "NF" place looking back.
But my parents didn't want me growing up in the NY public school system, understandably so, and I'm not sure if they wanted to/could pay for private school, so we moved to the Midwest, where my dad had gotten a job. I had a tough time growing up there. I wasn't like the other kids, the school was pretty much all-white (I was not), plus I was an INFJ male with a learning disability. To put it mildly, I didn't fit in very well at all. I was left out and even sometimes picked on; a couple kids especially, called me a monkey and tried to terrorize me every recess. I was a pretty tough kid, though, and I didn't stand for it. I fought back. In fact, that has been largely the story of my life, and why I picked my nickname. Certain types of people have tried to screw me over throughout life, and I had to fight back in some way and never let them get the best of me. They never did. While this seems like an admirable trait, it also resulted in certain issues with excessive defensiveness that I am just learning to get over.
Being tough and fighting back eventually stopped people from trying to pick on me, but it didn't stop them from leaving me out of their activities. Eventually, no one made jokes about me, but they left me completely alone. That was better than getting picked on, but it was still a tough and very lonely time. I always managed to find a friend here and there, though.
Middle school was for the most part, a terrible time for me. In 6th grade, I didn't understand the concept of turning in my work right away, so I failed pretty much all of my classes (except English where I got a B+ and Gym when I got an A just for showing up). I couldn't do math and science at all, so I just avoided the subjects altogether at first. I wanted to be popular and in the "in-crowd," but those kids had no interest in hanging out with me. I tried sitting with them at lunch one time and they just laughed at me. I never sat there again after that.
I went to middle school just before the highly-publicized school shootings, so I, like most all of the boys at school, was getting in fights all of the time (that I never initiated). The teachers did almost nothing to stop the fighting, except get me in trouble after the fact. They even had this ridiculous rule of, "If someone is trying to beat you up, don't fight back and you won't get in trouble." That was some of the most insane bullsh*t I've ever heard. Let some kid beat the snot out of me just so I won't get in trouble?! So, on top of getting bad grades and being left out constantly, I was labeled as a troublemaker too.
After my parents got me in line, I started doing a little better each quarter. Eventually, at the end of the year, I got all A's and B's for the first time in my life (and only time in middle school). It seemed like things were looking up.
In 7th grade, everything went way downhill. I had horrible teachers that year, and I mean HORRIBLE. My grades were equally horrible. In fact, my father, who ALWAYS blamed me when I had trouble in school no matter what, went to a parent teacher conference that year and was appalled at the level of the teaching. I transferred to a different school in 8th grade and did a little better there.
High school was when people and things just stopped making sense to me altogether. Before then, people were jerks, but they were at least honest about it. Young adult life was filled with phony and passive aggressive people. I thought some of the kids in middle school were preppy, but some of the kids in high school came from other schools were they were even preppier. I don't think I've ever felt more alienated than I did in high school. However, I succeeded academically, making the honor roll most semesters. I worked pretty hard back then. I still didn't really fit in anywhere, though. I even transferred to magnet school in 10th grade, which had a reputation for having "weirdos" and outcasts, and I didn't even fit in there! They still had created their own heirarchy of preps, "in" people, "out" people, etc!
In college, I hit my stride in many ways, and started to become successful professionally as well. However, even in my highest levels of success, I still never really felt like I fit in socially with the people I was in school with. I couldn't relate to them much at all. Sometimes people would become good friends, but some people would ignore and snub me. In fact, to this very day, old college "friends" come to my area of town and do not call me to hang out, but call others who were not even necessarily as nice to them as I was. To make it even more unbelievable, I've become pretty successful in my field and could open doors for them, but they still ignore me. At least they weren't fairweather friends, they were "any weather non-friends."
This kind of stuff used to frustrate me a lot, but I've learned from the few good friends I've had in life that true friends are not people you have to walk on eggshells with, or kill yourself to be their friend. You don't have to force with them. I've been very blessed to know those people in my life.
As far as my family life, I am unbelievably lucky. I have a few issues here and there...My ENTJ father pisses me off a lot sometimes because he constantly pushes my buttons. His side of the family is extremely f***ed up and are obsessed with gossip, being overbearing and setting an unbelievably high standard for their kids. However, I have usually been smart enough to distance myself from this stuff, and my Dad has also taught me a lot about reliability, hard work and manners. He's always been there for me, plus he has helped me financially quite a bit. My mother (INFJ) is an unwavering source of light in my life. She is tremendously uplifting to be around. I can tell her pretty much any problem I'm having, and she always knows what to do. She seems to know everything sometimes. I often think about how lucky I was in the parent department. About as lucky as I could be, honestly. My parents divorced when I was 9, but it never really bothered me that much.
One thing I've learned is that most of my personal issues have come from the outer world, and not quite so much from my family life. However, to answer the original question, yes: my environment has affected me...a lot. I've tried to find positives in most all of it, but no, you're not alone in your feelings.
Please remember what other posters have said. Never give up just because times get rough. Believe in yourself, because if you really want something, you can get it someday! Maybe life is harder for INFJs on average than some other types, but you learn amazing things from your experiences that give you a huge advantage in the long run.