An update:

We had a great long chat yesterday, and we are going slow it down for a few weeks and re-evaluate. He felt like I had expectations that he couldnt' meet yet and we need to get to know each other better. We will continue to see each other and get to know each other better to see where to go. It's a unique situation for me, but I open to it because this may be the point to set us up for long term success. It was very honest, and I was very caring and appreciative of him being him ( he was like " I move very very slow"), and his uniqueness. I think in a few days, he'll even realize more how rare of a person I am (in the best way!), to support us trying this.