That I come from old money.
It ceased being perplexing/annoying and is now highly amusing to me. Last person that brought this up told me (10 mins after she met me) that I come across as someone who "grew up in a gated community and spends their weekends at the country club".
I mean, WTF?
It used to really annoy me because it made me feel that I came across as pretentious. My Fi wasn't too happy with that.