Zimmerman didn't just need to feel threatened, he needed to be in reasonable fear of great bodily harm or fear for his life. While they were still standing Zimmerman had neither of these.
Once he fell and Martin continued to attack him while he was on his back, Zimmerman had a reasonable fear of both GBH and his life.
If Martin had been content to just break his nose and not ground and pound him, Zimmerman would be guilty of (probably at the most) manslaughter.
But Martin did follow him to the ground, straddle his torso and continue to punch him. That was the fatal mistake in this instance.