Remember a few episodes back, when Tate tells Violet that if she tells him to go, he will? He doesn't want to be there intruding into her life if she doesn't want him. I'm not sure what the "rule" is exactly; it's possible that the ghosts that go away choose to do so, some of the horrific interactions that occur seem incidental rather than planned by some of the ghosts.
That was such a nice tie-in.I also really liked Nora comforting Tate as a little boy, and him feeling the need to repay her kindness. Really like the actress who plays Nora. She's just right on the edge of looney-tunes.