Typing Mal is difficult, for a few reasons.
- Joss Whedon is a good writer, but he writes Fe characters way better than Te ones.
- The actors on any show will allow some of their own mannerisms and personality to shine through, often indicating a demeanor that doesn't square with the depicted type.
- Story writers don't generally write with MBTI type consistency in mind. They're more concerned with with making characters interesting than believable.
The main thing I get out of Mal from watching the show is a strong Te-style character, who makes decisions in a very Te way, who apparently thinks in a very Te way, but is played by someone who appears to me to be very Fe/Ti. It's as if Fillion has to try hard to be a stern captain at times.
I could see Mal as any of the following types:
ENTJ - ENTJs aren't necessarily very social, more concerned with organizing people (Te) than getting to know them personally (Fe). The intuition helps explain how his plans anticipate enemy actions so well.
INTJ - Not as likely as ENTJ, but he could just be a crusty INTJ stuck in a leadership position that he doesn't want, but for which he feels responsible (strong Fi values). He doesn't seem to be as lost in thought as an INTJ would be, however.
ENTP - Joss ends up writing some humor into the character that really doesn't fit the Te mold. The P fits the more adventurous nature of the character, and I think Fillion may very well be ENTP (not ENTJ as suggested elsewhere). I don't quite accept this in terms of the big picture, though. Even though he is often stuck in awkward adventurous positions, he doesn't particularly want to be in them. He does not want to rely on luck.
xSTJ - I won't deny the possibility of STJ, but the intuitive insight displayed by his character, especially in formulating plans, doesn't seem characteristic.
All of these, of course, need to be taken with a grain of salt. There is no real underlying personality with real cognitive functions beneath it all.
What I notice in the series is that Mal is written as a very "alpha" Te person. He does not use Fe-style management techniques to handle his crew. He is not particularly socially graceful, as an ENTP would eventually become at that age. He has kind of a trust-but-verify approach to handling his enemies, willing to stick to the terms of a deal until it is evident that the terms are broken. He doesn't seem particularly upset when he is double-crossed, but rather figures out how to recover from it. He doesn't chase down enemies, he ignores them as much as he can. He doesn't try to get even with enemies, and except for starting a brawl with Alliance soldiers in one episode, doesn't appear to want to settle a score. Rather than have a huge battle to protect the whorehouse in one episode, his first instinct is to take everyone off planet and avoid the fight entirely. In his epic battle with Niska, Mal humorously declares his lack of interest in finishing it all on his own. These are all Te approaches to me. His objection does not seem to be that things aren't worth fighting for, but that fights can go wrong, thus it is best to avoid a fight.
There are some indications of ExTP over ExTJ, namely that Mal doesn't appear to have a long term plan. I suspect that's more a matter of circumstance than design. It is, after all, and adventure story, and adventurers don't often have much time to formulate long term plans; it's often more a matter of surviving through each episode. In terms of his approach to daily matters, in terms of his approach to discipline on the ship, he is very TJ.