I wouldn't read a lot into it.
On Saturday, our neighbors out back threw a wild party and were making helluvalotta noise until 12-12:30am, so loud it sounded like they were in our living room. I was really irked, but at the same time I just rode it out... and if I had reacted poorly, it wouldn't have really been personal, it would have just been me being frustrated.
Basically, people can bang on walls for lots of reasons.
1. I'd feel bad I kept them up / annoyed them and apologize later.
2. I'd treat it like business, not personal.