If we assume Ns are 25% of the population as you say, if you aren't a complete hermit you know at least several. Those of us in more academic streams like me probably know a higher proportion of Ns, since they're probably overrepresented in universities.
It's really obvious you don't play sports, but there's actually a lot of strategy involved in playing most sports that isn't at all connected to the physical. Positioning, choosing angles of attack, timing, reading your opponents, predicting what they'll do and guessing the best counterattack, teamwork and communication, motivating, etc. The chess analogy is very apt except there's the added element of speedy decisions required. It's not just "durrrrr, kick that, hit that, now run over there". I'm not saying you think that, but it is coming across as a very.....simplistic idea of sports.
Also, many of my N friends are VERY involved in following trends for pro teams, predicting who'll win with what team and all that sports statistics stuff. I hate that stuff so I don't know what the appeal is, but it does seem abstract in a way (all numbers and patterns), even if it's tied to the physical.