I know other EFP men who are very popular and not considered soft or anything. They're usually funny, or good athletes/performers or something. There's also quite a prevalent stereotype of men being overgrown boys who can only drink, watch sports, and screw up fatherhood. That stereotype could easily be an EFP guy (though it could also be other types).
Furthermore, if you're a young Pe-dom with Fi, the Fi doesn't necessarily show up that much until you're older. I know when I was in high school, I was definitely thought of as more Tish (not that they knew types, people just thought of me as a math whiz/computer programmer). I was never very masculine, but now I am feeling a very strong pull from my Fi, to the point that some of my old guy friends don't relate to me as much as they used to. I've gotten spiritual and more feely (we're all connected, let's talk out our feelings, all that stuff), and they're still very Tish guys. They kinda take shots at me now, about being like a woman.
Being a feeler, doesn't make you necessarily effeminate. That being said, I think because feelers put more priority on connection, intimacy, and their own feelings, they are more likely to allow themselves to be vulnerable and open to others, which might cause them trouble as men.