my rule of thumb: if my dad says it, it's racist. this meets that test.
You haven't met my parents. :/
Being the ignoramus that I am, I actually have never heard of the term before, nor did I make the connection to realize what it meant. So to me, it did not sound offensive at first, but I didn't know what it was in reference to.