Human Compatible: Artificial Intelligence and the Problem of Control
Stuart Russell · 2019 CE
Excerpt
“AI systems are built by people and optimised for objectives that are imperfect; we should therefore design them to be useful and safe rather than perfectly aligned. (paraphrase of the 2019 book)”
Why the atlas keeps it
Russell argues that perfect value alignment is the wrong goal: systems are built by imperfect people, for imperfect objectives, and certainty about human values would be dangerous if it were achieved. The alternative is enough uncertainty about human preferences to keep a human in the loop as a genuine check. The excerpt below paraphrases the argument rather than quoting the 2019 book.
Terms this text reaches for
Ranked by the codex search index against the summary above, not hand-picked.
Normative uncertainty
ethicsUncertainty not about what is true but about what ought to be valued.
Goal misgeneralisation
alignmentAn internal goal that is correct for training cases and wrong just outside them.
Outer alignment
alignmentMaking sure the training objective itself encodes what was actually intended.
Meaningful human control
ethicsA standard requiring that a human have the time, information and authority to intervene.
Alignment
alignmentThe problem of getting a powerful system's goals to match what its designers actually meant.
Deceptive alignment
alignmentAn internal objective that is optimised to look aligned in the places anyone is checking.