It is not who is to say, it is what is to say.
There is a sense of society in which you can refuse to participate in it, or others may remove you from participation in it. Human society is a society of rational beings, and so your participation in human society is conditioned upon the extent to which you appear to you use reason. E.g., the rights of children are restricted because they've not yet developed the ability to fully use their reason, psychotics are locked up because they can't be trusted to behave rationally, the rights of criminals are restricted because they've shown that they've not used reason to understand what right action is, and people generally ignore persons they deem to be irrational.
So, you may decide the society you live in is largely irrational and so remove yourself from it, (e.g. you could move to a monastery, or a cabin out in the wilderness, etc.) On the other hand, you could deny the possibilty of rational action and stare at a wall until you die of starvation.
Aristotle said that man is a political animal: anyone who can live apart from society is either a god or a beast. Humans, by their very nature, must interact in order to realize their full potential as human beings. We are born ignorant and must be taught the good and the means to it. But teaching assumes insight, and insight is based on rationality. To borrow from Plato/Socrates: "the unexamined life is not worth living." To lead a thoughtless life devoid of rationality is to be less than human.
If you don't want to be a member of society, that's your choice. Maybe the society you live in is less than rational. I often think that the lunatics are in charge of the asylum. Or, you could choose to live like an animal. But, if you want insight into the good life, then it would behoove you to participate in a society of persons who seek to know what is good and the means to it, and who share their understanding with one another in order to increase the value and richness of life.