I have never had a problem engaging in small talk, or talk of any kind, as long as my relationships are formed in a relaxed manner.
I have different groups/individuals for different conversational needs:
I have a large group of S friends at my part time job, who I mostly joke around with and speak about sport; generally your typical everyday topics, but with humor.
I also have a large group of N friends at my University, where I get to speak more esoterically on a variety of subject matters which would send my S friends to sleep, in much the same way I imagine talking about sport would bore my N friends to death.
I enjoy both types of conversation, all though obviously, being an N, I am slightly more inclined to the N side of things.
"How dreadful!" cried Lord Henry. "I can stand brute force, but brute reason is quite unbearable. There is something unfair about its use. It is hitting below the intellect." ~ Oscar Wilde - The picture of Dorian Gray