All most all of Feynman's material is accessible to lay-people, even
QED, his own theory, is very accessible.
I also recommend
Six Not-So-Easy Pieces (also
Six Easy Pieces: Essentials of Physics Explained by Its Most Brilliant Teacher)
Not that this focuses on quantum mechanics, but you'll have an amazingly good micture of the "Standard Model" of physics after reading these.
However, as far as math goes. I suggest you learn about Hibert Spaces at least (just enough to interpret the inner-product style notation). It will save you a lot of trouble in the long run.