I think you're placing way too much importance on the "do things together" thing. While doing things together is cool. If we don't do things together, it's also cool, as long as you don't got pulling the "why don't we ever do things together, I feel left out and neglected" routine, when I do my thing, I wouldn't mind if you never shared any of my interests with me.
yeah, my istp is like this. we share some common interests, but not all. i do my theater thing and he likes to work out. but, we both like to go out and drink, so that is sort of when we spend time together. we both agree that what really matters is spending time with the person you love, not so much doing something you love. ALTHOUGH, his ex didn't share any common interests with him, and perhaps that's one reason she is his ex. i know it put a strain on the relationship. lucky me!
I am an IsTP, and my partner and I both have motorcycles, which is a big one. We also play video games together, which isn't as outdoors-oriented as I would like it to be but fun nonetheless.
I work out (weight lifting), am looking to get back into martial arts if I find a place that isn't too regimented with the ritualized elements and just gets down to the nitty gritty... but I would love (adore, enjoy, jump) it if my partner did martial arts or cliff diving with me. Getting to workout, get into the physically exerting "zone" of focus and being able to watch the person I'm most attracted to get sweaty?
Check for a good time.
It has become more important to me, because spending time with a partner in a way other than living in the same room (i.e., purposefully bonding & doing something together) seems to become different once ye live together and have been with each other for a while - you aren't specifically seeing each other once a week or whatnot to get to know each other, like in the beginning. Ye can end up spending a day or two within ten feet of each other but haven't actually spent any time with each other, if that makes sense. And I don't always feel like talking.
So... there are so many things to try, perhaps think of something you think ye'd enjoy and ask him if he would as well. Physical or mental, though I think physical is better if ye have the choice. You already mentioned sailing, so I won't make random suggestions.