I think this maybe an XNFP thing or particulary INFP, but I think we are
1) so wary of making other people uncomfortable
2) so aware of the emotional/social consequences of our actions and 'atmosphere'
3) put so much stock on people's perceptions of us
4) We are really hesitant to show people an 'ugly' side of ourselves or rock the boat.
I feel like this is a common growing pain for XNFPs and I've been where you've been (and I'm still at) so I don't mean it condescendingly. It's kinda the thorn in the side of many an XNFP.
Practical advice - for you it's more important to just say something explicitly and directly with no apologies and no gilding. Blunt. "I am angry at you because"
Those few seconds after you say it will seem like forever and you may feel such intense discomfort that you give into the urge to crack a joke or ligthen the tension but don't. Let the other person absorb what you have to say and respond. You will often find that they realize you mean business and will respect you for it.
I think in your particular case, it's better that you say something even in the heat of the moment that you may later "regret" rather than hold onto it and let it fester. Because if not in that moment, it will come out in other more destructive ways.
You need to retrain yourself to let things go and not second guess yourself. Give yourself permission to say something that other people don't like. Make peace with the fact that other people may react negatively towards you because you express displeasure or anger.
Once you give yourself permission for all that ^^ and are comfortable in your own skin when it comes to expressing anger, your presentation and energy and mindset will be so different that you will find you either get a better response or more importantly, you are more effective at communicating andyou feel better