I don't know you well enough to have any certainty about your type, but ISTJ actually crossed my mind once. Just from incidental observations it doesn't seem like you get offended easily at all. There is something that does seem consistent about you, but perhaps something else that isn't. When you mentioned that you can remember every detail of your home and where everything you own is, that does seem related to Si, but it could be something else. I haven't known INTPs or INFJs to have mastery over their belongings, but rather they have different ways in which their belongings are on the edge of being out of control. Most of the IN types attempt to exert control over their concrete domain, but tend to experience numerous failures in these attempts by losing things, hoarding things into random piles of confusion, organizing belonging only to have the disarray overpower the plan, throwing everything away and then regretting it, etc. Probably NTJs are the least out of control with "stuff".
I think it is difficult to know what type you are because you also have a connection to the intangible in a unique way. I'd be curious again to hear what types you have considered. I seem to remember INTP which resulted in some sort of upheaval, and then I've read INFJ on your profile. Somehow ISFJ doesn't feel quite like the right fit because you don't seem that sensitive to remarks by people.
You know, ISTP has quite a range of people who type as that. Both of their function pairs are the same as INFJ, just swapped, and there is a correlation to INTP as well, so I wonder if you could be an ISTP with strong Ni? Like in a Dom/Tert loop of Ti-Ni?