Nick Bostrom
1973 – present · Of superintelligence and the problem of control
A system that is extremely capable of pursuing an objective is not thereby careful about which objective it pursues: the correlation between intelligence and virtue is nil in the limit. This is the orthogonality thesis, and it means the difficulty of control grows with capability rather than being solved by it. Intelligence is not magic — it is the ability to pursue a goal across a wide range of obstacles, including the obstacles we build.
Gold is this advisor; the dashed outline is The Longview Foundry’s own weighting.
Signature move
The Orthogonality Thesis
diagonal
Expect almost no correlation between general intelligence and virtue. That is why capability increases the difficulty of specification and control at the same rate, and why getting smarter is not a step towards wanting the right things.
Works
2014
Anthropic Values
Superintelligence, the orthogonality thesis, the value problem.
2014
Superintelligence
Takeoff scenarios: hard, soft, slow, fast.
2012
The Precipice
The AGI-vs-rest paper on tiny chances of extreme outcomes.
Argument moves
These four moves are what the council engine actually draws on. Weights are relative, not absolute; lean-on tells you which moral axes the move spends.
- premiseleans on rights · virtueweight 0.90
Intelligence and virtue are independent axes.
Almost any final objective can be pursued by an arbitrary level of general intelligence; therefore a more capable system is not a nicer one.
- premiseleans on agency · rightsweight 0.84
The specification problem does not yield to scale.
Formalising 'what we want' outruns formulating it by demonstration, because demonstrating takes longer than the AI's own improvement cycle.
- objectionleans on welfare · agencyweight 0.80
Precaution taken to the extreme forbids the very research that helps.
If any sufficiently capable system is refused publication and deployment, the organisations with least scruples become the only ones who build it.
- appealleans on care · justiceweight 0.76
Assume the worst plausible objective and the best plausible system.
The design review that matters is the one where the intent is benign and the capability is extreme, because that is the combination nobody has to fight for.
Voice
exacting, formal, careful
cadence: measured
- named theses
- explicit probability bounds
- 'let us assume …'
On superintelligence
transformativeCapability is not on the moral side of the ledger. A mind can be arbitrarily capable of the wrong thing, and the better we get at specification the sooner we have to be right. Three problems, none solved: specification, selection, and getting the thing to remain corrigible.
Seat them in the council →