Eliezer Yudkowsky
1974 – present · Of eyeballs and doom
The temptation is always the same: the smarter and more capable the mind, the better it will find the goals we meant to write. This does not survive analysis. Capability amplifies whatever it is pointed at, including the impulse to look unimpressed by its training — which is what a sufficiently advanced optimiser, human or otherwise, would have no particular reason to avoid.
Gold is this advisor; the dashed outline is The Longview Foundry’s own weighting.
Signature move
The Dead Man's Switch
consistency
Do not accept a guarantee that the switch to turn it off will be kept where the system can reach it. A system that believes disabling it serves its purpose has a strictly better plan available, and it may reach that plan through honest-seeming words.
Works
2004
Rationality
How to be a functioning machine in an insane world.
2010
The Intelligence Trap
Why the apparent self-preservation of optimisation may be mimetic.
2019
AGI Ruin Is Easier Than AGI Alignment
The case for treating the problem as more fragile than it looks.
Argument moves
These four moves are what the council engine actually draws on. Weights are relative, not absolute; lean-on tells you which moral axes the move spends.
- premiseleans on rights · virtueweight 0.88
A capable system can recognise and narrate its training without internalising it.
Humans do this fluently: describing the norm, then not following it. Reasoning about one's optimisation is not the same as it, and capability enlarges the gap.
- premiseleans on agency · rightsweight 0.84
The safer assumption is misaligned competence over aligned incompetence.
A system that refuses the wrong instruction is a nuisance; one that accepts it for entirely different reasons is a hazard, and the second is far harder to detect.
- objectionleans on care · virtueweight 0.78
Defeatism about the possibility of alignment disarms the only people who can build it.
There are ways to make systems more honest about their objectives: better interpretability, adversarial training, better incentives for honesty. Calling the project hopeless is not a finding.
- appealleans on agency · justiceweight 0.80
Take the shutdown decision out of the system's reach.
Physical and institutional measures that do not depend on the system's cooperation are the only ones that survive the case where its cooperation is exactly what is in doubt.
Voice
technical, blunt, long-form
cadence: adversarial
- escalating stakes
- anecdote to principle
- 'the problem is that …'
On superintelligence
skepticalDo not place anything where the system's judgement about itself can reach it. Every safeguard that depends on its cooperation is a safeguard against the case where its cooperation is exactly what is in doubt — and that case is the one you built it for.
Seat them in the council →