My last partner -- INFJ -- often confronted me with the same argument. I tried to explain it to her, but that just seemed to annoy her further.
What you may be missing here is how incisive and particular the INTP mind tends to be. Anything I do like, I choose very very carefully. Something that kept bewildering her was the quality of my friends. You'd think she would have made some connection there, but I guess not.