It is insincere to say "yes" when you feel "no". Consider that you are not being genuinely polite or caring if you do things out of obligation only. This does not strengthen bonds with people, but it actually weakens them, as interactions are not founded on real feeling. People don't feel security, loyalty and care when things (including time and energy) are given in this manner. It all becomes a meaningless charade.
To be a truly giving person, you must do so wholeheartedly. Every time you consider saying "yes" when you feel "no", realize that you are lying to people and that they deserve better than that. They deserve your honest "no".
I've never had a problem with saying no, in most situations (pity dating is whole other story), and it's because I know the above deep down. I know it's not kindness or thoughtfulness, but cowardice and passiveness. People don't appreciate it in in the long-run, even if they may not see how the dynamic it creates in relationships is related to this behavior.