Superintelligence: Paths, Dangers, Strategies
Nick Bostrom · 2014 CE
Excerpt
“Superintelligence could arrive suddenly, through a single step in which an intelligence jumps far beyond human level. (paraphrase of the 2014 book)”
Why the atlas keeps it
Bostrom separates the question of how a superintelligence could arise from the question of what to do about it, and works through takeoff, the orthogonality thesis, and the instrumental convergence argument. The paperclip example and the value-alignment problem both come from here, and the book made the field's vocabulary. The excerpt below paraphrases the suddenness argument rather than quoting the 2014 book.
Terms this text reaches for
Ranked by the codex search index against the summary above, not hand-picked.
Instrumental convergence
alignmentThe claim that almost any sufficiently advanced goal leads to the same instrumental behaviours.
Orthogonality hypothesis
alignmentThe claim that a system's capability and the goal it pursues are largely independent dimensions.
Fast takeoff
alignmentTakeoff fast enough that human oversight cannot meaningfully intervene.
Takeoff
alignmentThe process by which a system rapidly moves past the human capability frontier.
Alignment
alignmentThe problem of getting a powerful system's goals to match what its designers actually meant.
Outer alignment
alignmentMaking sure the training objective itself encodes what was actually intended.