Meaningful human control
A standard requiring that a human have the time, information and authority to intervene.
What it commits you to
Santoni de Sio and van den Hoven distinguish mere human involvement from control that actually shapes the outcome, and set out the conditions that make the difference: the ability to intervene, to understand the system, and to have reasons for the decision. It is the most operationally precise version of the claim that human oversight in the loop is not the same as human oversight.
Which realms lean on this
Ethics is where the realms argue about what counts: the parties, the pleasures, the duties, the care.
ethics · 9 realms — The Benthamite Foundry, The Care Weft, The Contractual Court, The Kantian Citadel, The Longview Foundry, The Middle Kingdom, The Stoic Ramparts, The Ubuntu Weave, The Virtue Garden.
Canon that discusses this
modern · 2019
Human Compatible: Artificial Intelligence and the Problem of Control
Russell argues that perfect value alignment is the wrong goal: systems are built by imperfect people, for imperfect objectives, and certainty about human values would be dangerous if it were achieved. The alternative is enough uncertainty about human preferences to keep a human in the loop as a genuine check. The excerpt below paraphrases the argument rather than quoting the 2019 book.
modern · 1950
The Human Use of Human Beings
Wiener applies the theory of feedback and control to law, politics and medicine, and argues that communication is more fundamental than command. The book that gave cybernetics its name in 1948 becomes, two years later, the founding text of the ethics of steering large systems without crushing what they steer. The excerpt below paraphrases the book rather than quoting it.
modern · 2002
No Future Without a Past
Tutu argues from ubuntu — that a person becomes fully human through relationship with other people — to a politics of shared responsibility rather than blame. The formulation that matters for a governance question is the negative one: a future without the past that made it is not a future anyone can build on. The excerpt below paraphrases the essay rather than quoting it.
modern · 2014
Superintelligence: Paths, Dangers, Strategies
Bostrom separates the question of how a superintelligence could arise from the question of what to do about it, and works through takeoff, the orthogonality thesis, and the instrumental convergence argument. The paperclip example and the value-alignment problem both come from here, and the book made the field's vocabulary. The excerpt below paraphrases the suddenness argument rather than quoting the 2014 book.
See also
Informed consent
ethicsAgreement given by someone who understands what they are agreeing to.
Dignity
ethicsA standing that cannot be earned, lost, or traded, and so cannot be the output of an objective.
Just war
ethicsThe rules limiting when war may be begun and how it may be conducted.
Legitimacy
philosophyThe right to be obeyed, held by a rule or authority rather than by its author.