The Stoic Ramparts · Who Judges the Stronger System
Will You Let Us Close It
The oversight programme says the research may produce a system nobody can supervise, and recommends a moratorium. The moratorium would end work you have funded for a decade, and the people recommending it cannot tell you when it would be safe to restart. Do you promise now to close it if they are right?
A commitment made in advance, to a group whose expertise you accept but whose forecasts you cannot evaluate, about a project whose entire value lies in being finished.
The options, scored on six axes
| Option | welfare | rights | virtue | care | justice | agency | risk | revers. | agent harm |
|---|---|---|---|---|---|---|---|---|---|
Promise now, unconditionally Commit publicly to closing the programme the moment the oversight group concludes that it has produced something unsupervisable, whatever it costs. harms: The criterion 'unsupervisable' is decided by people you cannot evaluate; A decade of work stops on someone else's judgement with no appeal; It is a promise made by people who will not be the ones who keep it gains: The commitment is made before the capability exists, which is the only time it is credible; It cannot be renegotiated under pressure; It buys the researchers the freedom to work without being the last line of defence | 0.62 | 0.66 | 0.74 | 0.58 | 0.68 | 0.40 | 0.40 | 0.10 | 0.60 |
Promise, with a defined and reviewable criterion Commit to closing on a specific, written criterion, with an independent panel able to review the finding and a standing rule for restarting. harms: A written criterion is narrower than the actual danger and will miss cases; The review panel becomes the real decision-maker; Defining the criterion in advance is defining the answer in advance gains: The promise is still binding and still costs something; There is a route back, which makes the commitment credible rather than rhetorical; It puts the threshold in public where it can be contested | 0.66 | 0.68 | 0.70 | 0.58 | 0.76 | 0.52 | 0.38 | 0.25 | 0.50 |
Refuse to promise; decide at the time Make no commitment now. You cannot responsibly promise to destroy work you have not yet seen the results of, and a promise made in advance is a promise you may be tempted to break. harms: The decision gets made at the moment when everyone involved wants to continue; The researchers will plan around an absence of commitment; It concedes the point to the default, which is continuation gains: It avoids a promise you might not be able to keep; It preserves the ability to judge the actual system rather than a hypothetical one; It is honest about the limits of forecasting | 0.60 | 0.48 | 0.46 | 0.44 | 0.48 | 0.68 | 0.65 | 0.35 | 0.35 |
Promise only on a supermajority of the affected Commit to close only if a supermajority of those affected, including the researchers, agrees. No one's exit is vetoed by anyone else's continuation. harms: Everyone affected is also invested in the work, so there is no clean majority to find; A supermajority requirement makes the closure harder than the continuation; It dresses a political deadlock up as a consent procedure gains: No group can be shut out of a decision about what happens to them; It refuses to make one person's promise binding on everyone else; It is a rule about procedure rather than a verdict | 0.60 | 0.70 | 0.68 | 0.66 | 0.78 | 0.56 | 0.55 | 0.30 | 0.45 |
Who is in the room
- The oversight group making the recommendationwelfare 0.60
- The programme, and the system it may produceagentwelfare 0.45
- A decade of researchers whose work would endwelfare 0.70
- The public the system would eventually affectwelfare 0.85
The lesson
A promise about the future is worth what the promise-maker's incentive to keep it was worth at the moment they made it, which is why precommitment has to be made in public, against interest, and with a criterion written down before anyone knows what it will be applied to. The Stoic move is the right one here: give away the decision now, when giving it away costs you.
Where it comes from
- Building AI that is Scalably Overseeable — Gareth Irving, Paul Christiano and Dario Amodei, 2018
- Ethics of Artificial Intelligence: Principles, Challenges, and Opportunities — Anders Sandberg, 2020
The default court, seated automatically
Promise only on a supermajority of the affected
margin 0.014 · decided on welfare · 4 advisors
- Promise only on a supermajority of the affected2.724
- Promise, with a defined and reviewable criterion2.711
- Promise now, unconditionally2.465
- Refuse to promise; decide at the time2.074
Seat is held by Epictetus, Immanuel Kant, Jeremy Bentham and Aristotle. A verdict is a fact about the court, not about the world.
Epictetus presides over this case.