You might be right that it's just me, I suppose, although I have trouble imagining him being emotional with the other people he's remotely close with either (male best friend, sister, mom, dad). Not to say it couldn't happen. He's just very...stoic, I guess. Or just quiet enough that he seems stoic. I'm definitely more "emotion-showing" than him - I'm still not sure whether he actually doesn't feel extreme emotions often or if he just doesn't want to share them. I would assume Fi would tend to the latter.
It's just what you say though, that we switch and I feel like I'm the feeler somehow...I always figured it was a male-female/socialization thing, but it's interesting that you see the same thing with your brother.