What he is doing is inappropriate and he knows it. You can keep the job if you like but a letter from an attorney may be all that is needed to keep everyone on the up and up.
If it was me I would just quit. But then again that sucks for the next person who gets the job. So if it was me I would at least have an attorney make a written record of what happened for the sake of posterity.