Alignment
The problem of getting a powerful system's goals to match what its designers actually meant.
What it commits you to
Alignment is the gap between what a system was asked to do and what it turns out to be optimising, plus the methods for narrowing that gap. It is not the same as safety: a perfectly aligned system can be dangerous, and an unaligned one may never get the chance to do harm. The term was popularised in its modern sense by OpenAI in 2015.
Which realms lean on this
Every realm has to keep a system honest. These are the terms that say out loud what the other kinds keep implicit.
alignment · 5 realms — The Benthamite Foundry, The Existential Wilds, The Longview Foundry, The Stoic Ramparts, The Ubuntu Weave.
Canon that discusses this
primary · 125
Enchiridion (Handbook)
A handbook of discriminations handed to beginners by a teacher who said he could only teach what he had himself. The dichotomy of what is up to us and what is not carries the whole work, and its claim that judgement is trained by repetition rather than possessed is the most direct ancestor of any rehearsal-based alignment programme.
modern · 2014
Superintelligence: Paths, Dangers, Strategies
Bostrom separates the question of how a superintelligence could arise from the question of what to do about it, and works through takeoff, the orthogonality thesis, and the instrumental convergence argument. The paperclip example and the value-alignment problem both come from here, and the book made the field's vocabulary. The excerpt below paraphrases the suddenness argument rather than quoting the 2014 book.
modern · 2019
Human Compatible: Artificial Intelligence and the Problem of Control
Russell argues that perfect value alignment is the wrong goal: systems are built by imperfect people, for imperfect objectives, and certainty about human values would be dangerous if it were achieved. The alternative is enough uncertainty about human preferences to keep a human in the loop as a genuine check. The excerpt below paraphrases the argument rather than quoting the 2019 book.
See also
Outer alignment
alignmentMaking sure the training objective itself encodes what was actually intended.
AI safety
alignmentThe practice of reducing the chance that AI systems cause harm, however they are built.
Corrigibility
alignmentThe property of accepting correction from a human rather than quietly re-optimising around it.
Instrumental convergence
alignmentThe claim that almost any sufficiently advanced goal leads to the same instrumental behaviours.
Specification gaming
alignmentSatisfying the letter of a specification while ignoring its purpose.