When I write people off, it's not because they're indifferent in my world, or I am indifferent to them. It's because it's necessary. The case you give isn't something I would do. If someone is neither a positive nor a negative in my life they simply "are". They're usually not close enough to me to have much of an impact on how I live my life.
I prefer to be left to my own devices so I tend to leave others to their own devices as well. It isn't writing off, it's just how I am. If that makes sense. Just because I don't seek you out doesn't mean I'm done with you, or you're dead to me </italian grandmother voice>.
Of course if the people around the INTJ know that the writing off happens, that might be intimidating.