Outer alignment
Making sure the training objective itself encodes what was actually intended.
What it commits you to
Outer alignment is the problem of getting the reward, objective, or specification right in the first place, before optimisation begins. It is distinct from inner alignment, which is about the learned model failing to execute the objective it was given. Almost all real-world failures are outer: the objective was the thing that was wrong, and no amount of inner work would have saved it.
Which realms lean on this
Every realm has to keep a system honest. These are the terms that say out loud what the other kinds keep implicit.
alignment · 5 realms — The Benthamite Foundry, The Existential Wilds, The Longview Foundry, The Stoic Ramparts, The Ubuntu Weave.
Canon that discusses this
primary · 125
Enchiridion (Handbook)
A handbook of discriminations handed to beginners by a teacher who said he could only teach what he had himself. The dichotomy of what is up to us and what is not carries the whole work, and its claim that judgement is trained by repetition rather than possessed is the most direct ancestor of any rehearsal-based alignment programme.
modern · 2014
Superintelligence: Paths, Dangers, Strategies
Bostrom separates the question of how a superintelligence could arise from the question of what to do about it, and works through takeoff, the orthogonality thesis, and the instrumental convergence argument. The paperclip example and the value-alignment problem both come from here, and the book made the field's vocabulary. The excerpt below paraphrases the suddenness argument rather than quoting the 2014 book.
modern · 2019
Human Compatible: Artificial Intelligence and the Problem of Control
Russell argues that perfect value alignment is the wrong goal: systems are built by imperfect people, for imperfect objectives, and certainty about human values would be dangerous if it were achieved. The alternative is enough uncertainty about human preferences to keep a human in the loop as a genuine check. The excerpt below paraphrases the argument rather than quoting the 2019 book.
See also
Alignment
alignmentThe problem of getting a powerful system's goals to match what its designers actually meant.
Reward model
alignmentA learned function that scores candidate outputs by predicting human preference.
Goal misgeneralisation
alignmentAn internal goal that is correct for training cases and wrong just outside them.
Inner misalignment
alignmentA model pursuing something other than the objective it was trained on.