I think most NFs do this (especially xNFPs) and this brought me to one conversation I once had with an ENFJ friend when discussing about this issue:
"You should not remain neutral. There is a time in everyone's life when you must take sides. By remaining neutral when others are using your neutrality to do wrong, you become involved with their bad behavior. Take sides and don't stay on the fence--that is not loyal to anyone."
In workplace, I almost always try sitting on the fence to avoid dramas. And it frustrated some of my co-workers (I was a team leader). I suppose by remaining neutral when people have an argument based on different opinion is the best thing to do. But now when I think back, that's also another way to lose loyalty from people who are close to you.
I wonder if how other NFs think of this?