You do have a point that type is free from certain social constructs that are usually used in person perception. I also agree with you that there is some truth in stereotypes. Stereotypes are necessary and useful because they provide us a way of gaining quickly information about someone when are resources are limited. However, as you suggest, it does not make sense to use them in situations where we have the resources to gain more fine grained information about someone, such as with people close to us.
Your anecdote with your ex highlights some problems that could arise from using the MBTI. People excel at finding evidence that supports a hypothesis they have formed and tend to disregard evidence that is contrary to it (confirmation bias). For example, someone who believes that "F"'s are not capable of logical thinking might see multiple instances of logical thinking by an "F" as only exceptional, and use a sole instance of illogical thinking as "proof" that confirms the initial suspicion that an "F" person cannot think logically. Also, people's expectations about other's behaviors might cause them to behave in such a way as to situationally provoke the expected behavior in another (self-fulfilling prophecy). For example, by acting in a warm, trusting and affectionate way with a person who we think is an "NF", there is a good chance that we will sollicit these behaviors in return regardless of the actual type of the person. These natural biases in human functionning may make it more difficult for someone into the MBTI to be able to see the more specific personality traits a person may hold regardless of his or her type.