Instrumental convergence

The claim that almost any sufficiently advanced goal leads to the same instrumental behaviours.

What it commits you to

Instrumental convergence argues that self-preservation, goal-content preservation, and resource acquisition are useful for nearly every objective, so they arise from almost every objective. This is why unaligned systems are expected to look similar across projects. Critics note it is not a theorem and that the behaviours it predicts are often already covered by ordinary safety work.

Which realms lean on this

Every realm has to keep a system honest. These are the terms that say out loud what the other kinds keep implicit.

alignment · 5 realms — The Benthamite Foundry, The Existential Wilds, The Longview Foundry, The Stoic Ramparts, The Ubuntu Weave.

Canon that discusses this

modern · 2014
Superintelligence: Paths, Dangers, Strategies
Bostrom separates the question of how a superintelligence could arise from the question of what to do about it, and works through takeoff, the orthogonality thesis, and the instrumental convergence argument. The paperclip example and the value-alignment problem both come from here, and the book made the field's vocabulary. The excerpt below paraphrases the suddenness argument rather than quoting the 2014 book.

Instrumental convergence

What it commits you to

Which realms lean on this

Canon that discusses this

Superintelligence: Paths, Dangers, Strategies

See also

Alignment

Orthogonality hypothesis

Empowerment

Takeoff