I think there's a couple different things that can contribute to this. First, I agree with @Seymour that anyone can get drained if they end up spending too much time dealing with others, especially introverts. Outside circumstances like work, family obligations or whatever can force people to spend more time interacting with others than is optimal for their temperament, and that makes you a person feel worn out.
I have wondered if IxFJs may have more of an inner push toward taxing themselves socially compared with other introverts, because of their drive toward engaging others all the time and meeting people's needs and whatever it is FJs are wont to do. It does seem to me like there would be more push-pull coming from the inside with you guys.
I've known a couple IxFJs who would avoid or put off simple activities like going to the store or passing by a neighbour on the way out, because they didn't feel like socializing much and felt guilty about that, like they should be really friendly all the time. Don't know how much I should tie type into things like this, as both of them had some assertiveness problems...but I really can't imagine myself worrying much about things like that. No matter how neurotic I can get in other ways.