The primary appeal is his ability to give his characters surprising depth, and the depth of his characters is leveraged to create some remarkably intricate plots that just seem to flow from the base characterization. In the meantime, he can entertain viewers/readers by making his characters interact with each other in very human ways.
There are very few writers who do this well. David Eddings was one such, in my opinion. Other than that, especially in fantasy or sci-fi, the characterization is often a weak point, often reflecting the cognitive biases of the author. It takes great insight into human nature (I would type Whedon as INFJ) to be able to build characters that hide the authors cognitive biases very well.
I find that I usually need to watch an entire season of one of his shows before really understand his characters and appreciate what he's doing with them. Out of context, his episodes will often lack the special charm that most people see in his work, because he doesn't reveal that much of any character at any one time, but rather lets you get to know them by seeing them in many different contexts/stories/episodes. This is, by the way, how he hides his cognitive biases from his audience: he portrays the behaviors very accurately, but only lightly brushes in the attributed motivations. Thus, seeing/reading his work out of context, you see people doing stuff that either makes obvious sense (because the reasons are immediate to what is happening) or make no sense at all or only superficial sense (because the reason was indicated in three other episodes you've not seen). His really great Buffy episodes, such as Hush or the musical episode, for example, simply aren't as enjoyable if you haven't spent time on other episodes and thus understand the Buffy characters.