Have you tried giving him time to prepare for the "serious" conversations? I know for me I usually need quite a bit of time to process things. I need to feel prepared before I can speak. What about next time saying to him, "I really feel like we need to talk about such and such. Can we discuss it tomorrow evening over coffee?" It might help if he feels like he has enough time to prepare himself first.
I love INFPs but one thing I've noticed is that sometimes I am surprised by the serious conversations. Somehow I never see them coming. It takes me awhile to adjust before I can sort out my thoughts enough to respond.