I too trained for years. And I sometimes wonder if that confidence has actually PREVENTED fights from happening. Whenever someone tries to test me now. I pretty much go into fight mode (this doesn't mean that I start throwing bombs, just tense up and get ready). Strangely, it's usually around this time that the "barker" usually diffuses.
I had a guy slap me on the ass very hard in the bathroom once at a drunken volleyball party, and I shrugged it off, he did it again, I turned around and slammed him into the wall and stuck my elbow into his throat and just said "Careful." I have no memory of even meaning to do this, it was just a reaction. He did nothing after that.