Really, I think you need to ask him how he would like to solve the problem. It is *his* problem, after all. He's the one not fitting in with everyone else. If you try to solve it for him, he will probably resist. If you "reach out" clumsily, he will withdraw. It really has to come from him to have any chance of success.
One approach that might be worthwhile is to frame it as a problem in his area of expertise: how would he counsel another family with this problem? Give him enough distance to analyse it intellectually, without getting overwhelmed by all the emotional baggage. If he approaches it from this angle, it has a better chance of getting under those defences he habitually erects to keep him inside his comfort zone.
When he is engaging with you enthusiastically about something he's passionate about
: music, for example, take care not to crush that enthusiasm by being overly critical, or dismissive, or impatient. 5s aren't terribly resilient when it comes to that kind of rejection. We expect not to be understood or accepted - when you offer confirmation of that, we quickly determine it's not worth the effort to even try.
I would guess that the best chance for establishing a closer bond with your Dad is by finding common intellectual interests that you can explore together, one-on-one. He'll get something from going with the family out to dinner/the beach/walking/whatever, but primarily, we live in our heads, and that's where we like to be engaged.