It depends on the person and how / in which situation they are saying it.
When it is a person which I don't value as close to me, I don't care or am a bit angry that they make judges on me which they aren't able to do.
A person close to me where I know that is saying it only in a funny way I usually have no problem with, maybe laugh about it. I know I am actually lazy in many things, so it is ok. But when they say it in a very negative way, it obviously might bother me.