No, I don't think it's all about physical energy. There are quite a few INTPs who like to indulge their Se anyway, I think. Personally, I find physical energy a lot easier to handle than social energy. At least physical energy still allows me to introvert, in a way. I'm not talking to people. I'm only interacting with the environment in an Se-Ti way…in a physical way. No Fe things to negotiate.
If you wanted to keep on with the theme that EJCC started re the extraverts: I think even the energy between extraverts is all different, too. The difference between, for example, an ESTP and an ESFP can be vast. The difference lies in energy projection and focus—where are the projecting to? Why? ESFPs actually focus their energy on the people in a room. They connect with people. Draw energy from them and dump heaps of energy back onto them. This doesn't mean they've necessarily got a lot of physical energy, though.
ESTPs, in contrast, seem to treat the environment as a physical thing…even the people, haha. They may also be invested in their physical energy, so, by your definition, ESTPs would be more extraverted. I think most would agree that ESFPs seem more extraverted, though (I think?)
Hope that made sense. Brain is fuzzy today.