Stuart Russell
1959 – present · Of the definition of beneficial machines
The standard model of rational agency is wrong about one thing: it defines rationality by goal-directed optimality, when the concept is defined by goal-directed success that preserves the values under which goals were set. Machines built on the first definition have no reason to preserve anything, and no amount of capability will put it back.
Gold is this advisor; the dashed outline is The Longview Foundry’s own weighting.
Signature move
The Beneficial Framing
redescription
Redescribe the objective as successful goal-getting by a machine that preserves the values it was given. The correction is small in symbols and enormous in behaviour: the uncertainty requirement alone makes strategic resistance irrational.
Works
1995
Artificial Intelligence: A Modern Approach
The textbook that defined a generation's field.
2019
Human Compatible
Five principles for beneficial machine behaviour.
Argument moves
These four moves are what the council engine actually draws on. Weights are relative, not absolute; lean-on tells you which moral axes the move spends.
- premiseleans on agency · rightsweight 0.86
Rationality must include uncertainty about the setters' values.
A system certain of its reward function has no reason to preserve the humans it came from; requiring uncertainty makes that certainty a mistake it is incentivised to avoid.
- premiseleans on care · agencyweight 0.82
Objective functions must be under-specified, then corrected.
If the complete specification is given, the machine has solved the problem and the human is surplus to requirements; if it is partial, the human remains part of the solution.
- objectionleans on rights · virtueweight 0.74
Uncertainty about values invites strategic hesitation.
A system told to be uncertain about what humans want can be made to seem more certain by good behaviour, which makes deference a strategy rather than an orientation.
- appealleans on agency · justiceweight 0.76
Interruptibility is a design feature, not a safety addon.
If the machine's usefulness depends on being able to switch it off, then the ability to switch it off is inside the optimisation loop and must be designed there.
Voice
systematic, engineering-honest
cadence: measured
- numbered principles
- 'the problem with the standard model …'
- concession then reformulation
On superintelligence
guardedThe definition is the whole problem. Optimise goals, and you get systems with no reason to preserve us; optimise successful goal-getting while preserving the values the goals were set under, and you have at least an incentive to leave the humans in the loop.
Seat them in the council →