Aptitude is not necessarily related to type. Let's face it there are quite a few INTP's out there who are not very good at math. But INTP's who do have an aptitude for math are superb at it, because Ti is the best function for doing pure math (and Ne helps too). I know plenty of INTJ's that are also very good at math, but they use Ni and Te. INTJ's always seem to struggle with math if they don't understand how it applies to the real world because their logic uses Te, but even though Te is their secondary function they can be superb because their Ni picks up a lot of the slack. In other words in some capacity Ni can mimic what Ti does naturally. Also an INTJ who does a lot of math will naturally develop their Ti.
I think what has happened in your case is that your apptitude for math has either developed your Ti or else you are simulating it with Ni. So along with your normal Ni and Fe being your two most developed functions you also have Ti in there as well. You end up applying logic to your Fe which is not what people do very often (or to the extent that you are explaining), so you have some unusual sounding behavior.
Also based on your description I would guess that you are an extreme introvert. Is that an accurate assumption? For example there are plenty of moderate introverts who can seem extraverted in some situations, and likewise moderate extraverts who seem introverted in some situations. You sound to me like an extreme introvert. (This is not to pass judgement, I am simply trying to accurately assess the situation.) But that is what I would expect from someone with both a strongly developed Ni and Ti.