I actually used to think I was an INFP, so yeah, they're similar enough to mix up. Both are quite shy, quiet and subtle in communication, and both are difficult to read or get to know. As mentioned, both are also fairly zany when they want to be, and have weird streaks in taste and behavior. Both can come off as cold and arrogant to people who don't know them well, and both have similar problems fitting into social scenes. They are both fairly non-confrontational and laid back. So there is a lot in common between the two.
I think, when manifested externally, the differences tend to include the following;
-INFPs don't make fun of people too much, even if the person in question is hilarious or ridiculous in some undeniable way. INTPs will have no qualms about making fun of people.
-INFPs will give you that "that's not right" look when you do something they consider bad, and then they won't tell you anything more about it. INTPs, OTOH, rarely disapprove, either in their minds or externally.
-INFPs become more sweet when you get to know them and talk to them, and they like you enough to be comfortable opening up around you...INTPs do not. In other words, you will never observe an INTP and have an "awww" reaction to their behavior, whereas with an INFP that is very possible.