It doesn't have to be off-the-wall to be an idiom. "The term idiom refers to a set expression or a phrase comprising two or more words. An interesting fact regarding the device is that the expression is not interpreted literally. The phrase is understood to mean something quite different from what individual words of the phrase would imply. Alternatively, it can be said that the phrase is interpreted in a figurative sense."
Idiom - Examples and Definition of Idiom
A couple of decades ago, people were commonly using the phrase in a more literal sense, saying that they weren't comfortable asking a teacher to review a grading decision, or that they wouldn't be comfortable letting their kid fly in an airplane alone. They meant that they experienced a sense of inner unease at those ideas that was telling them that they didn't want to do them.
In the past ten years, I almost always hear the phrase used in a different sense, the one I described in the OP. It has nothing to do with a sense of unease telling them that something might not be a good idea. They just say it because they think it sounds too direct if they just say, "I don't like it."