Pretty much, yes. To continue with the black box analogy, here is more or less how I see myself:
* At the center, there is this undestructible black box.
* Around it, there's this huge layer of puffy matter. It's made up of my own home-grown stuff, but also of stuff imported from the outside: expectations, appreciation, and so on. (That's what Cascade explains so well in her post, for example.) Press upon it from the outside, and you can easily mold it, punch holes in it, cut through it, whatever.
* On the outside, there is the crust, the walls put up to protect the puffy insides.
Without the walls, anyone, and I mean *anyone* could pass by me and punch me, cut me, hurt me, deeper and deeper and deeper. The only thing that would eventually stop them would be when they reached my black box core. But that's a LOOOOONG way down!! The damage done on the way would be of monumental proportions!! I'd be a total wreck.
So there's no other solution but to build and maintain those walls, and let in only the people I can trust to not poke and punch too much around.