That's because we don't like being the subjects. We like to be the observers not the observed.
It reminds us that we're just the lowly humans like everyone else, not the impartial panel we like to think of ourselves as. I know that when I think, usually about global issues, that I imagine it from the point of view of an alien who has no interaction with the planet. We have trouble thinking of ourselves as part of the multitude. We're never in the crowd, the crowd is around us.
It is arrogance, and there really is no excuse we can give for it.