In other words, yes, there is accountability to the voters, but that's easily assuaged by telling the voters what they want to hear. There's more compelling accountability to everyone to whom you might owe favors, or on whose good graces your fortunes depend. This list could include figures from business and industry, the judiciary, or the bureaucracy itself. It would also include those above you in the political chain of command through either government or party channels.
For political appointees, there is no accountability to voters, but only to one's political patron(s). For career civil servants embedded in the bureaucracy, there isn't even that.