I recently turned 30. My dad knows I watch anime, which to him is cartoons... So I kind of hear that from time to time.
Don't worry. I'm almost 25 and I watch anime. My close friends are in my age range (23-27) and we go to anime conventions. That's far more geeky and childish than just watching it. Though, I only watch anime in intervals. Basically, until I burn out. Then I break for months.