I hear what you're saying, for sure, and I have an infj friend who I feel expects interactions/events to play out a certain way, else she won't be very happy - so I feel in a sense she is setting the mood/bar and I need to comply. That said --- I have another infj friend who isn't this way at all. Not all infj's are the same.
Re 'tracking' -- I honestly think it's not entirely conscious. I think it's partly conscious, but I think it's more just an automatic thing they do; it just happens. They mentally note something. They can't un-note it once they note it, I don't think. I think it's about seeking consistency in people --- and if they realize there's a pattern of the other person always being 10-30 minutes late, they'll come to expect that and that's set for them, now -- they've defined the person in that way. Then they decide if they're ok with that or not. And so on. I think.
How would they feel if other people did that to them? Maybe the thought doesn't automatically cross their mind, but sometimes I find myself thinking that INFPs and INFJs are one in the same.
I guess at a human angle it's like sure, you know what to expect from each of your friends (and family members). -- but I don't let that cloud my opinion of them. It's not like their bad or dramatic side ways out their good side.
But if it's true that INFJs want perfection, then one bad or dramatic thing could axe a hole in the wall and then what I'm seeing is that there's no attempt to repair that hole. It's just left open and that person is inside of the hole trying to patch things up and the INFJ is like no, this can't be fixed.
I guess I'm in the realm of positivity overcoming negativity. Maybe that's why I adore the heroes in everything and my friend seems to side with the under-dogs? Who knows.