tinkerbell's probably got a point that the female>male transition's also causing issues to a degree on top of such, however, I think the ENTP>ISTJ thing may be covering it a bit as well.
I can't speak for him personally, but I can speak for myself. I know very well, that I will wall off very fast if someone tries to push in too hard too fast. If yeu don't prod me at all, I will *NEVER* open up, as I have absolutely zero reason to do so. If, however, yeu push me too forcefully, it'll actually cause me to wall up even more than if yeu'd said nothing at all. There's a very delicate balance, that's honestly not in the middle, it's much closer to the 'nothing' side, and this may be a similar case that yeur mother's facing.
I'd request that yeu explain to her that, if she's inquisitive, to be very careful with wording, and specifically watch reactions. If she notices him STARTING to become brief, or terse in replies, or won't answer something directly or skirts the issue, DO NOT PERSUE IT FURTHER. If he is even remotely similar to me, it will probably just make him feel very uneasy and scared, and will react by refusing to say anything. Nudge it a little bit at a time, keep a small pressure going, but don't push when they're not comfortable with it, they'll just resent yeu for it.
Personally as well, I've noticed that at certain times, I'll be significantly more open... just certain timeframes and mindsets where I may just be more likely to explain myself in detail, and won't be as restrictive against being nudged. These happen to me virtually always late at night when my mind's not 100% aware, and the emotions may take over a bit better, though I'm not sure if there's any correlation between my doing that and anyone else, it may be better to try to prod around nighttime than during the day.
The basic idea though, is let him go at his own pace, think of it like someone walking along a very narrow path, they're terrified of falling off it, kinda like a tightrope or something. If yeu don't push them at all, they're not going to move because they're scared to. If yeu push too fast, they're going to cling tightly to it and refuse to move because that's even worse than standing still. Slow and steady will work much better more likely.
That being said, the 'always is vague' response is typical for most ENTP's that I've seen, seems to be a method of exploration... leave it vague to start, develop a broad, generalized understanding, then set the limitations around it in a boarder to see how far the concept reaches, then worry about the inner workings.
In an emotional context, and the fact that he's an E, chances are he seriously doesn't understand his own emotions very well. Having someone who can understand them better would help a great deal, but they have to do things not so much with the mindset of 'learning' about him... because he doesn't even know the answers himself likely, since an ENTP learns primarily by explaining themselves through someone else, so they can't really put long hours of solitude style thought into their inner workings; instead if she looks at it as a matter of trying to help him understand himself, and having her understand more in the process, then the 'vague and generalized' descriptions are just the starting point to begin with, if she can help him understand himself from that vague and generalized starting point, then he can move outward to defining the boarders, the extreemes and such, and only after he's comfortable with those, will he be likely to work inwards to understand major concepts of himself and the working as a whole. It's a slow, agonizing process, but well worth it in the end.
That, at least, is how I know for a fact that I work over time, and from the limited description given, it seems like it may fit his issues with such. Get yeur mom to try that method, and see if it works any better. Or better yet, copy/paste this response and ask him if he thinks it makes sense to him or if he thinks it would work. It may or may not be accurate to him as it is to me, but he'll likely be interested in the concept and read it just out of curiousity, and will likely want to think about whether it actually affects him in that manner, it may or may not, but he'll probably at least try to see if it fits, and even if it doesn't, chances are, with an outline of possibilities, he can be persuaded to go through things one at a time and list whether he thinks each applies or not.
