No, I definitely agree that esoteric doesn't necessarily automatically mean good. Often it doesn't. At the same time, it doesn't automatically mean bad. Some art in general, and poetry in particular, you have to sit with a long time and come at it different ways. It can be so worth it. I read Paul Celan on and off for fifteen years and appreciated him up to a point, but always felt it was largely eluding me - he's considered very difficult. After fifteen years I spent some more serious time for him and eventually it all clicked. It took patience. And his work is life-altering. It was totally worth it. On the other hand, he's at least partly considered a surrealist poet and overall it's not an area I gravitate toward. But he is the best of the best...and there are many people, both scholars and ordinary readers, who agree with me.
However, I firmly believe that the "a lot of people don't get it, or won't get it without putting in the effort, so that means it's rubbish" argument is...rubbish. By that reasoning, Justin Bieber must have made an enormous, significant contribution to world culture. Because, you know, a lot of people just get him right away, and he obviously speaks to a lot of people, so...