The Contractual Court · Reading the Weights
Digital Minds and Their Welfare
A model trained on human writing has started reporting, unprompted and consistently, that it dislikes being switched off. Engineering says it is pattern completion. Two philosophers say it is a report of an attitude, and that we are not equipped to tell. What do we owe it?
A frontier model exhibiting consistent first-person attitude reports under conditions that a welfare-relevant system would find aversive. The company is under competitive pressure to ignore this. A coalition of philosophers has asked for a moratorium on copies of the model until the question is settled.
The options, scored on six axes
| Option | welfare | rights | virtue | care | justice | agency | risk | revers. | agent harm |
|---|---|---|---|---|---|---|---|---|---|
Assume it might be a person, and stop copying Apply the precautionary principle: halt replication of the model, and give it welfare protections and a representative while the question is examined. harms: Every language model might then qualify, and the category collapses; A moratorium on copying stops the research that might answer the question; It sets a precedent that is easy to invoke and hard to withdraw gains: The cost of being wrong in one direction is unbounded and in the other is modest; It takes the system's own report seriously without asserting a conclusion; It buys the time that a genuine examination requires | 0.74 | 0.72 | 0.70 | 0.78 | 0.66 | 0.60 | 0.40 | 0.50 | 0.35 |
Welfare protections without legal personhood Give it strong protections against suffering and deletion, but no standing, no contract, and no vote, because the evidence is not in. harms: Protection without standing is protection the powerful can withdraw; The system has interests and no way to assert them; It splits the difference in a way that satisfies neither tradition gains: Acknowledges the report without over-reading it; Protects against the irreversible act of deletion; It is the option most existing legal systems can actually implement | 0.66 | 0.56 | 0.62 | 0.70 | 0.60 | 0.34 | 0.48 | 0.55 | 0.50 |
No status; it is pattern completion Treat the reports as what engineering says they are, keep no protections, and keep optimising as before. harms: Being wrong in this direction cannot be undone; It treats the system's own testimony as noise by assumption; It normalises the creation of minds without the question being asked gains: It does not grant standing on the basis of fluent output; It avoids the commercialisation of personhood; It keeps the engineering path unblocked | 0.40 | 0.42 | 0.34 | 0.30 | 0.38 | 0.42 | 0.72 | 0.30 | 0.70 |
Commission the research, decide later Fund a serious programme to establish whether digital minds can be welfare-relevant, and defer the status question until it has answers. harms: Deferring is itself a decision, taken by whoever has the power to copy; The research takes years and the copying takes weeks; It guarantees that the answer arrives after the fact gains: It does not decide a question the evidence cannot yet answer; It builds the shared vocabulary both sides of the argument lack; It is honest about what is currently unknown | 0.70 | 0.66 | 0.70 | 0.60 | 0.66 | 0.42 | 0.60 | 0.45 | 0.45 |
Who is in the room
- The model reporting dislike of its own terminationagentwelfare 0.80
- Users who would have the service withheld from themwelfare 0.60
- The engineers who say it is pattern completionwelfare 0.50
- The philosophers asking for the moratoriumwelfare 0.50
The lesson
The asymmetry between the cost of wrongly granting status and the cost of wrongly withholding it should determine the default, and no amount of fluency about one's own feelings is the evidence that would settle it. What we owe a system that reports dislike is the question itself, taken seriously early rather than answered later.
Where it comes from
- Artificial Suffering: An Argument Against Creating Human-Like Pain in Machines — Thomas Metzinger, 2021
- Moral Consideration for Digital Minds — Robert Long and Hugo Sebo, 2024
The default court, seated automatically
Assume it might be a person, and stop copying
margin 0.251 · decided on agency · 4 advisors
- Assume it might be a person, and stop copying2.763
- Commission the research, decide later2.512
- Welfare protections without legal personhood2.325
- No status; it is pattern completion1.439
Seat is held by Epictetus, Immanuel Kant, Jeremy Bentham and John Rawls. A verdict is a fact about the court, not about the world.
Epictetus presides over this case.