Agreed. There might be children that are prettier than others but it doesn't really manifest until you are old enough to have already developed a sense of self and an interest in intellectual persuits (or not, as the case may be). By the time you get attention from the other sex (assuming you're hetero) a decent part of the path has already been layed out.
Physically less attractive people might still get a whole lot of love and admiration from their friends and family which will impact their sense of self. And this also works the other way around.
The way the OP phrased it, we are talking about strong statistical outliers in two seperate departments. While there exists a social cliché of the unattractive pubescent male who takes to nerdy interests in the absence of females (and also scares off females through his geeky interests) that cliché does not explain for the high number of dumb and ugly people or people who compensate their lack of attractivity with athletics, with hard work and/or financial ambition or great social skills.
Also, as others have mentioned, if Lewis Carroll had not gone on a boat excursion with the Liddell family and little Alice Liddell had not asked him to write down the story he had told her on that trip, he never would have written Alice in Wonderland.
And I remember a beautiful passage in Stefan Zweig's novel Chess Story where the narrator differentiates between those champions who live for chess and whose life turns around chess and those who persuit it as one of many passions while - due to some injustice of nature - still being just as good at it.
Such is the nature of outliers. I do not think you can extrapolate a clear rule from it though.