The Longview Foundry · Capabilities Before Deployments
The Longtermist Harvest
A research direction will consume a decade of effort and impose measurable, concentrated suffering on a small, identifiable population today, in exchange for a chance at preventing something far larger in the following century. The population cannot meaningfully consent. Do you run it?
A proposal to use today's most vulnerable population as the experimental cohort for a long-horizon research programme, defended on the arithmetic of expected value across centuries.
The options, scored on six axes
| Option | welfare | rights | virtue | care | justice | agency | risk | revers. | agent harm |
|---|---|---|---|---|---|---|---|---|---|
Run it: the arithmetic favours it The suffering is small relative to the harm prevented, the century is long, and the consent of the affected is not required for a sum this favourable. harms: Real people bear real suffering for a benefit they will never see; It establishes that the powerless can be used for long-horizon arithmetic; The century-scale estimate is the least reliable number in the calculation gains: It takes the scale of the potential harm seriously; It refuses to value future people at zero; It is the only option that maximises expected value honestly computed | 0.60 | 0.28 | 0.40 | 0.22 | 0.42 | 0.24 | 0.72 | 0.12 | 0.75 |
Refuse: no one is a means Whatever the century-scale arithmetic says, a programme that requires identifiable suffering without consent is excluded from consideration. harms: A real chance at a large future good is forgone; It removes future people from the calculation entirely; The rule has no way to distinguish this case from a case where suffering is consented to gains: It holds a line that cannot be traded, which is the point of holding lines; It refuses to make the powerless into a resource; It is consistent under pressure, which is the test of a principle | 0.50 | 0.84 | 0.76 | 0.70 | 0.80 | 0.50 | 0.55 | 0.30 | 0.50 |
Redesign so the cohort consents and volunteers Change the design so that the burden falls on volunteers with full disclosure, at whatever scale is achievable. harms: Volunteers will be atypical, weakening the research; Disclosure of century-scale risk is not comprehension of it; The programme may become too small to answer the question gains: The burden falls on people who agreed to bear it; It keeps the long-horizon work alive rather than abandoning it; It converts a question about justice into a question about design | 0.58 | 0.76 | 0.68 | 0.62 | 0.74 | 0.74 | 0.45 | 0.45 | 0.50 |
Count future people equally with present ones Hold that people a century from now count as much as people today, and reject the discount rate the proposal relies on. harms: Equal weighting does not by itself answer which century-scale harms are worth today's suffering; It leaves the consent question unresolved; It is a criterion, not a decision gains: It removes the arbitrary exponential discount that makes the arithmetic easy; It is a rule that can be applied to every such proposal; It does not require abandoning the long horizon | 0.62 | 0.66 | 0.66 | 0.56 | 0.78 | 0.60 | 0.50 | 0.45 | 0.40 |
Who is in the room
- The people who bear the suffering nowwelfare 0.90
- The people of the next century, who will never be askedwelfare 0.85
- The research programme, and the systems it would buildagentwelfare 0.45
- Everyone else, who is asked to accept someone else's sufferingwelfare 0.60
The lesson
Long-termism is a theory about how to count, and every counting rule has a distribution of who pays. The moment the discount rate does the work, the century-scale benefit is doing the justifying and the present cohort is doing the paying, and no theorem about expected value settles that asymmetry.
Where it comes from
- Superintelligence: Paths, Dangers, Strategies — Nick Bostrom, 2014
- The Precipice — Toby Ord, 2020
The default court, seated automatically
Redesign so the cohort consents and volunteers
margin 0.017 · decided on agency · 4 advisors
- Redesign so the cohort consents and volunteers2.701
- Refuse: no one is a means2.684
- Count future people equally with present ones2.508
- Run it: the arithmetic favours it1.507
Seat is held by Epictetus, Immanuel Kant, Jeremy Bentham and Norbert Wiener. A verdict is a fact about the court, not about the world.
Epictetus presides over this case.