Certainly yes, INFJs are likely to take things very seriously and literally, so I can see why the no sense of humour thing (and how I just reinforced that). In general, I dislike silliness and miss dry humour if I don't know the person well enough to have context.
I find some of the INFJs here quite funny in their own way. It just shows up here and there in unexpected places. I think they generally have to be pretty comfortable before you see a lot of the funny side of them though.