It has a lot to do with perception. Just like I mentioned in the intelligence thread I know I'm above average in certain areas, but so are many people. Someone might not be very book smart, but know how to fix and repair everything in your house AND your car. Another person seems literally mentally retarded in elementary school, but then grows up to win the Nobel Prize for Physics. I'm hesitant to vote myself above average in light of all of that.
Same with physical appearance. I know I was considered a very pretty little girl, and in high school I got a great deal attention from boys at my school, and was jokingly voted one of two girls Most Likely to Pose for Playboy my senior year. I obviously also consider myself attractive enough to do things like be an exotic dancer. All that said, I'd clearly be above average, right?
Well, yes, and no. Leaving my small Southern town and moving to places like Las Vegas and L.A. make me feel much more average. By the standard of beauty in the media at the moment, I'm still probably considered on the attractive side of average, but not above average. I think doing things like being a stripper exposes you to so many pretty women or women with great figures that you begin to really think of yourself as less extraordinary.
I tend to compare myself to the largest number of people as possible, and that's why I voted average.