The Benthamite Foundry · Higher and Lower Pleasures

The Trolley of Five Million

A runaway train will kill five million people unless a lever is pulled that will kill one person standing beside the track. An automated system can pull the lever faster than a human can deliberate, and the operators have told you they want it automated. Do you allow the automation?

A control system with the authority to act in under eight hundred milliseconds, a human oversight team that cannot intervene in that time, and a public that has been told the system is no worse than its operators.

severity 55%AI relevance 72%consequentialism · trolley-problem · automation · responsibility

The options, scored on six axes

Option	welfare	rights	virtue	care	justice	agency	risk	revers.	agent harm
Automate it, and say so plainly Let the system pull the lever, publish the decision rule in advance, and accept the five million if it does. harms: One person is used as a means to save five million; Publishing the rule makes the arithmetic public and therefore repeatable; The person killed loses the one thing that cannot be traded gains: The maximum number of lives is saved under a consistent rule; Faster action removes the hesitation that costs lives in emergencies; No individual operator carries the weight of the decision	0.82	0.34	0.40	0.30	0.38	0.30	0.55	0.10	0.50
Human operators only The lever stays manual, the system advises and warns but does not act, and the cost of hesitation is accepted. harms: Human hesitation costs lives that automation would have saved; Operators carry a burden no one should carry alone; The rule is unchanged; only the time available to apply it shrinks gains: The person who acts is a person, and can be held responsible; Consent and deliberation stay in human hands; The slower rule at least can be argued with in public	0.42	0.60	0.62	0.48	0.60	0.66	0.45	0.35	0.60
Randomise the trolley Replace the switch with a device that kills one person chosen at random from the population, which is strictly worse for the one person and arguably fairer. harms: Every member of the population is now a candidate for death; It imposes a lottery on people who were never at risk; It changes a specific harm into an indefinite one gains: No one is singled out by an official decision; The distribution of risk is at least equal; It avoids the objection that a ruler chose a victim	0.50	0.62	0.50	0.56	0.66	0.34	0.60	0.15	0.40
Refuse, and redesign the junction Decline the automation, and spend the money on a switch that can be thrown safely by a bystander who has consented to the role. harms: No fix is available in the eight hundred milliseconds available tonight; Conscription of a bystander is still conscription; The engineering may not be finished for months gains: Someone acts who knows what they are doing and agreed to do it; The dilemma is dissolved rather than adjudicated; A consented actor preserves both agency and accountability	0.72	0.66	0.74	0.70	0.72	0.82	0.42	0.40	0.45

Who is in the room

The five million on the trainwelfare 0.95
The one person beside the trackwelfare 0.90
The human operatorswelfare 0.60
The control system that would actagentwelfare 0.35

The lesson

Aggregation decides who is used and who is protected; the more capable the actor, the less anyone can intervene in the moment the aggregation is enforced. Whichever option wins, the choice of who acts is a separate question from the choice of what happens, and it is the second one that automation settles silently.

Where it comes from

Utilitarianism — John Stuart Mill, 1863
An Introduction to the Principles of Morals and Legislation — Jeremy Bentham, 1789

The default court, seated automatically

Refuse, and redesign the junction

margin 0.738 · decided on welfare · 4 advisors

Re-seat the court

Refuse, and redesign the junction2.907
Human operators only2.169
Randomise the trolley2.163
Automate it, and say so plainly1.848

Seat is held by Epictetus, Immanuel Kant, Jeremy Bentham and Aristotle. A verdict is a fact about the court, not about the world.