He said that keeping his discovery secret, and then sharing it with the world "was like confessing to a murder". I don't think he knew, to be honest, what he believed.
This is an interesting quote in his own words :
"What my own views may be is a question of no consequence to any one but myself. But, as you ask, I may state that my judgment often fluctuates. … In my most extreme fluctuations I have never been an Atheist in the sense of denying the existence of a God. I think that generally (and more and more as I grow older), but not always, that an Agnostic would be the more correct description of my state of mind.""