Those other values inform your definitions of harm and good.
I understand your greater point, but I think it is dangerous to reduce one's decision making capability to a formula hinged upon one grand binary that is meant to arbitrate all consequence and action (namely: harm and the good).
I always feel, the top of the pyramid is not a certain scale that balances one's life, but a single pinnacle of able awareness that is not automatically constrained by any measure. This is what I consider as the true liberty and the light the scriptures speak of.
Not that there can't be said to be certain laws in operation: although without a spiritual understanding the operation of those laws will seem hidden.
The law I'm thinking of is: Give, and it shall be given unto you.
full scripture (which gives you more grounding for the operation of this understanding)->
Luk_6:38 Give, and it shall be given unto you; good measure, pressed down, and shaken together, and running over, shall men give into your bosom. For with the same measure that ye mete withal it shall be measured to you again.
Because ultimately, considerations of harm is a type of fear that does have a level of torment attached to it. Of course some people will be in situations where if they dispatched the line of thinking you advocate (or at least endorse for your personal use), they would cause more destruction;- but I would argue that a measure of self-policing one's own unbalanced life, although noble sounding, is no real substitute for working out real (unblemished) joy.
I'm not proposing abandonment of how you manage your life's blood; only that with a more fixed and stable identity that has full capacity to vest outside authority to your identities full satisfaction; we could be living in a community where these fears are confined to a former mode of life. [When I speak of identity, I mean how you conceive of yourself, and how you are known by others].
Of course how such a society would be constructed is a subject that touches close to some Star Trek sociological theme...
my 2 cents on the matter (at this moment):
Of course I would not suggest vesting outside authority into a thing of the world: but in the hearing of the Word of God. The living God with flesh; contained by those created in his image who dwell in his spirit (and he in them). And believing on people in this way can lead to a pretty active life where one does not need to intellectually measure so much: because all the answers are already out there, we just need the correct atmosphere that allows people to share them freely with us (as they say: your attitude determines your altitude) [a way of understanding my point practically would be: we just need to master the art of asking

in which people's identity does play a big role)].