Did you ever tell your coworker, "Hey, can I get back to you later? I've got a lot of work to do. " instead of going the passive aggressive route? I'd let them talk for a few minutes and then interject that before the conversation goes on too long.
People come to my desk all the time and I just tell them that in a friendly tone and they say no problem and disappear. I know most jobs have peaks and valleys and certainly your coworkers can understand that. I guess I'd think to myself that if someone comes over to chit chat with me frequently and I've not had negative interactions with them they must like talking to me or something. Some people just rove around the office when they feel restless, it may not be anything they're doing to purposely aggravate you or keep you from doing your work. Just be up front with them.