For some reason, some people maintain quite a high level of self-confidence and certainty of themselves even when their caregivers inadvertently undermine it; instead of trusting the caregivers opinion, they trust their own. (So they respond to their own faith within themselves.)
People who are impacted negatively by the caregiver's opinion are impacted only because they permit it and because the opinion matters to them so some inexplicable reason.
Yeah, true.
But I was actually referring to infants and very young children, as they completely lack perspective and coping strategies to govern their own self-esteem, which is first and foremost influenced in the first relationship humans have. That relationship, sadly, influences and leaves a mark to later interaction and relationships we have, as well as to the basis of our self-esteem. (But I don't mean we couldn't later on affect to it in any way

)
I do understand how it works for older children, that they're able to for instance maintain their own self-esteem despite the negative feedback they get of themselves, but self-esteem is not being born by itself.
When the child grows up and develops perspective and begins to understand not everything they encounter from their parents/peers is a face-value, they can start objectively reassessing their perceptions and change their view about themselves and take in new information as well

And children have remarkable talent to find positive feedback (mirroring relationship, where they get positive mirror for themselves) ; if it's not from their parent, they'll search for it wherever they can, from their teacher, friend's parent etc.