I think the term "racist" has been watered down over past few decades due to over-usage. 50 years ago, a racist was a person who wanted to treat a particular race as a second class citizen. This would either happen because one race would feel superior to the other, or because one race felt threatened by the other. For example, the primary reason why the Nazi propaganda worked so effectively was because most people felt economically threatened by the Jews. On the other hand, you will find that most of America and Europe, at one time, literally felt that Africans were a lower species. Therefore, they did not deserve the same rights as "humans".
Today, we know that differences among races are minimal to a degree that there are overlaps between individuals. Therefore, all races are much more equal than we once thought they were.
We are not all racists, because today most Americans don't want to treat certain races as second class citizens. Those who do are ignorant bigots or attention whores.