I guess what's most important is maintaining a consistent weight. Each person have their own set point "ideal" weight range. If you're getting somewhere around 2000 calories a day with a good variety of food choices and you still see no difference in your weight then I wouldn't worry too much.
Perhaps your doctor was concern about your recent weight drop. BMI charts aren't useful for somebody who's fine boned but has a high metabolic rate.
I think it's recommended that you have about 1-2 teaspoon of oil/fat every day? Calcium is important though... substitute things like nuts and fortified orange juice/soy etc if necessary... also lactose tolerance can be improved if you drink more milk. Just need to take it slowly.