Wind-Up Rex already nailed it, lol.
However, I have a thought through MBTI terms, along the lines of what Marm said. It sounds like you took time in your life to develop your Fi side, but you have yet to integrate it with your Te. The tertiary can be a challenge because it's very on-off, everything or nothing.
Integrating Fi and Te is an art of learning how to have a well of deep compassion and being able to reach out and connect that to someone else, while using Te as a method of detachment to prevent yourself from getting so lost in your emotion that you can't deal objectively with the situation. I don't pretend like I have anywhere near mastered the art, myself. Wind-Up Rex talked about fear – understanding when to lean heavy into Fi processing and when to lean heavy into Te processing can help mitigate fear, because Te is essentially fearless, while Fi has deep knowledge of the emotional void of fear. I think of Fi as the deep well of compassion inside, and Te as the force shield outside. You can lower and raise that guard to protect yourself and others, or to expose yourself and others.
I think eventually the goal would to be so compassionate and so strong that you can protect yourself and completely expose yourself at once – Wind-Up Rex's Warrior's Heart. It's about understanding that you are at once everyone and no one, about recognizing that you can understand and tap into the emotions that drive people to do illogical things but recognizing that you can transcend those emotions if you choose. So you would recognize things as having the potential to bother you for ____ reason, but also being able to remove yourself from that personal emotion and see the situation for the simple exchange that it is. Transcendence, if you will. And forgiveness for everyone who doesn't live up to the ideals and principles you wish to see, yourself included.I don't mean that something bothers me and I just keep it inside. I mean literally for stuff to not even register as something that would bother me.
You also sound like you might be an Enneatype 1 – such a strong focus on principles, and how yourself and others “should” be. Exploring that system might be useful.