I can almost appreciate actual colonialists more, since they fully admitted to what they were doing and were unapologetic. Nor did they really think they had "rights". Rights are always won or taken. Rights are just ideas, not real in and of themselves. It's another word for strength. Or rather, a way to deflect from it. To sterilize it.
The difference between natives is that they got their asses kicked. Not that they're indigenous necessarily. And in the current age, people kind of feel like shit about it, saying "Ease up on the poor bastards. We've done enough."