huh. (ENFJ-wise) It really doesn't sound like he is that into you, and if he is, he's suppressing it. I'd think it over long & hard. Figure out what you can & can't accept in a relationship with him. He seems to be trying to make sure the relationship is not exclusive, even if he hasn't done anything with anyone else. If this is not okay with you, say something now. If you don't want to be too direct, ask if by saying "not serious" he means seeing other people is okay? That doesn't directly imply that you think he will or that you want to.
Once you get that cleared out of the way (if it bothers you at all to be unsure), if you are happy for the most part, I'd be as patient with it as you can. As an ENFJ, I've dated people I didn't like very much but I was fairly obvious about it. While what he says doesn't really convey strong feelings for you, it doesn't seem to deny them either. He could be suppressing them to deal with whatever is going on elsewhere in his life.