I tend to feel for the people who are spacey, introverted, and socially "off" at times who are easily misinterpreted, so I was trying to think how one might distinguish the real thing. I would say that passive-aggression would tend to be connected to an attitude of arrogance, withholding with a sense of superiority? Also, a simple approach would also be to determine if the person has a reason to be angry in that instance, although some people are more laid back and don't get upset by the same things as others.
The first man to raise a fist is the man who's run out of ideas. H.G. WELLS
The first principle is that you must not fool yourself, and you are the easiest person to fool. FEYNMAN If this is monkey pee, you're on your own.SCULLY
There has to be intent; the only problem is intent is not always conscious. So a person that is passive aggressive is (by definition) displacing anger, but may or may not be consciously aware they're doing so.
I find that when I'm passive aggressive I'm not very aware (or sometimes I'm aware that I'm angry at the person, but don't realize it's unwarranted by their current actions). It usually takes either someone calling me out or me having enough time for some reflection to realize what's happened.