It is true that we impose our perceptions on the world, yet it still seems to me that we share the same hallucinations. At least we see enough similarities to say that we have the same physical objects in front of us and the ideas that we discuss could easily be kept in perspective by both of us.
What about mathematics, as Galileo said it is the language that God wrote the universe with. Perhaps it symbolizes how everything works in the universe, and there are many other ways to do this. Hence this is an example of how a code for the way the world is structured could be immanent within nature and outside of our heads. With Kant's transcendental idealism the world as we experience it is subjectively conditioned, but nonetheless the noumena, the real thing must exist objectively as that is not subjectively conditioned, we look at that and translate it into the world of phenomena.
It may be true that we are not creatures to know the objective truth, yet again this in no way precludes it from existing as the world behaves not in accord to our preferrences but in accord to the laws of nature that do not have to agree with what we like or what we don't like.