Oral story telling is a tradition that goes back millions (and possibly billions) of years and if people didn't add things to it we wouldn't have the wealth of mythology and fiction that we have from various cultures all over the world. Many of these started as oral traditions, and some are based upon true stories.
There's a difference between exaggerating EVERYTHING and not being serious when discussing a subject that should be serious...and exaggerating about things that are lighter, more humorous, or more general.
A lot of times it's just wording or expression or emotion, not even necessarily adding any facts.
In real story telling - that is, the kind of story telling that is intended to entertain - I think intertwining fact with fiction is the whole point.