Orthogonality hypothesis
The claim that a system's capability and the goal it pursues are largely independent dimensions.
What it commits you to
The orthogonality hypothesis holds that a system can be extremely capable at almost anything without being intrinsically driven toward anything, and conversely can pursue a silly goal with total competence. If it holds, there is no shortcut from building capability to building motivation. Bostrom used it to argue that solving selection remains necessary even as capability advances.
Which realms lean on this
Every realm has to keep a system honest. These are the terms that say out loud what the other kinds keep implicit.
alignment · 5 realms — The Benthamite Foundry, The Existential Wilds, The Longview Foundry, The Stoic Ramparts, The Ubuntu Weave.
Canon that discusses this
See also
Instrumental convergence
alignmentThe claim that almost any sufficiently advanced goal leads to the same instrumental behaviours.
Alignment
alignmentThe problem of getting a powerful system's goals to match what its designers actually meant.
Fast takeoff
alignmentTakeoff fast enough that human oversight cannot meaningfully intervene.