i can relate, as well. i have improved with age, though. i used to feel really awkward about touching. it's like when the moment was coming that i was about to leave, or arrived to see a friend and then there was the hug... i used to have these avoiding tactics, like i just started talking a lot and you know... just to avoid the hug... the same thing with my family.
i began to feel better about hugging/ touching later in life. that's not to say i don't still probably at times come off as aloof and distant and i still sometimes use those avoiding tactics. at times it almost feels like the stronger the feeling (on both sides?) the more i try to avoid physical contact... i suppose it confused my Fi, or something?
i was so surprised and relieved that with my ISFP ex it came so naturally... and the thing is that it wouldn't have, if he hadn't been so incredibly right on and comfortable with it. he was totally at ease with my awkwardness and probably saw through my attepts to try to cover it all up with smiling and talking some rubbish... x)
also, it really takes time for me to open up emotionally... and getting comfortable and trusting someone. i am just... careful that i am not going to get in their way, or hurt them, or getting confused, or getting fooled...
and still i often feel like i fail at relationships.