If any resemblance INFP-INFP helps:
I would never use that phrase with a guy that I'm not attracted to. I would be too scared of confusing and leading him on (emotional gore).
On the other hand with the guy I'm attracted to... yes (hoping that he will see what the phrase means).
And when I do the smug in a situation like that inside I am like:
OMG, OMG, SHIT, SHIT, SHIT, oxygen, oxygen, aaaaaa, breathe, where is the damned water (a pleasant emotional overload).
OR: really, do you see it like that, I'm freaking disappointed but i'm going to keep my face straight (handling a blow).
SMUG usually means a revelation with huge emotional impact. Whether the impact is positive or negative can be recognized by emotions accompanying the smug: wild happiness, little sweet awkwardness, sadness, anger...
And did he add "But..." after the phrase?