I would imagine athiests are easier to dislike based on the misconception that they come across as a bold-faced "NO". They can be a wall to people who want the landscape open to eye up all of the air. There are a lack of commonalities between atheists and people who are religious, or agnostics who simply shrug.
Contemplate how a great deal of humanity needs to stake itself down with a security blanket. People who tear themselves from the security blanket are misconstrued and intimidating. Either that, or easily dismissed and looked down upon. The people who cling, desperately, are threatened by one's ability to hang onto life without a support that is above and beyond them. The people who turn their nose up are the one's who feel their security blanket is fact and omitting such a fact will besmirch their truth.
There are middle grounds, but balance is difficult to come by. This is especially true of the topic of religion. This also comes from people not wanting to take the time to understand, or who have a bad brush with someone who doesn't understand it for themselves.