It certainly will drive the price up, but then any used car bought from a dealer will cost more than the same car bought from a private party. I have bought half dozen used cars, always from private parties, and have never had a problem. Dealers have a reputation for not being forthcoming, and doing all they can to get more money from the customer. With an individual, it is easier to sense whether they are being forthcoming, and you are more likely to get an honest accounting of the car's history and any problems. Yes, you don't get a warranty, but check any dealer warranty very carefully as they try not to promise you anything that would cost them much should you need to invoke it.
So, I recommend you do your homework on car make/model/year. Look for cars with good owner satisfaction and repair records over 5-7 years. Look at websites like Edmunds and Kelly Blue Book. Check out cars advertised by people in your local area. If the seller doesn't seem on the level, or you feel at all uncomfortable, move on. If you find a car you like, you can always have a trustworthy mechanic look it over for you before making an offer. They will probably charge a small fee, but that is a much better deal than the dealer markup. If you have a friend who knows something about cars, or who has purchased used in the past, enlist their help as well.
Good luck!