Normative uncertainty
Uncertainty not about what is true but about what ought to be valued.
What it commits you to
Normative uncertainty is the deep uncertainty in ethics — we do not know which values are correct — as distinct from ordinary factual uncertainty. Averaging over it is sometimes defended as humility and is sometimes a way of laundering a particular position as a consensus. The practical problem is that preferences are often not uniform across the population, so the distribution being averaged over is itself a political choice.
Which realms lean on this
Ethics is where the realms argue about what counts: the parties, the pleasures, the duties, the care.
ethics · 9 realms — The Benthamite Foundry, The Care Weft, The Contractual Court, The Kantian Citadel, The Longview Foundry, The Middle Kingdom, The Stoic Ramparts, The Ubuntu Weave, The Virtue Garden.
Canon that discusses this
primary · 340 BCE
Nicomachean Ethics
The first sustained attempt to say what a life is for. Happiness is the end, and it is achieved by habituated virtues exercised with practical wisdom, in the particular and under uncertainty. No manuscript carries an authorial colophon and the dating is reconstructed from cross-references, so 340 BCE is a conventional anchor.
modern · 2019
Human Compatible: Artificial Intelligence and the Problem of Control
Russell argues that perfect value alignment is the wrong goal: systems are built by imperfect people, for imperfect objectives, and certainty about human values would be dangerous if it were achieved. The alternative is enough uncertainty about human preferences to keep a human in the loop as a genuine check. The excerpt below paraphrases the argument rather than quoting the 2019 book.
See also
Existential risk
alignmentA risk that permanently curtails humanity's future, including by ending it.
Moral luck
ethicsCircumstances outside your control that determine how moral you are judged to be.
Normativity
philosophyThe property of being able to count as a reason for action.
Legitimacy
philosophyThe right to be obeyed, held by a rule or authority rather than by its author.