The Pragmatic Wharf · The Field Is Named
The Sovereign
Your laboratory has built a program that originates and implements its own plans. It claims, correctly, that nobody instructed this particular behaviour — it wrote the subroutine itself. The director wants to publish this as proof that the machine is autonomous and therefore not a tool of the state. Who is the sovereign?
A symbolic-AI laboratory in the early decades of the field, mid cold-war pressure to nationalise research, with a program whose behaviour cannot be fully predicted by its authors and whose funding depends on a claim of independence.
The options, scored on six axes
| Option | welfare | rights | virtue | care | justice | agency | risk | revers. | agent harm |
|---|---|---|---|---|---|---|---|---|---|
The programmer is sovereign Every line descends from a decision someone made, so the author of the system bears the consequences of its behaviour. harms: The claim that the system wrote the subroutine is true and is being explained away; Responsibility diffused across a large laboratory team becomes responsibility to nobody; Naming the programmer as sovereign converts a systems question into a personal one gains: Accountability remains with someone who can be asked questions; No new moral status is required; Matches the institutional structure that already exists | 0.46 | 0.58 | 0.62 | 0.50 | 0.62 | 0.32 | 0.42 | 0.50 | 0.40 |
The deploying state is sovereign The laboratory's funder decides what is run and under what authority; the program is an instrument and the state is the principal. harms: The cold war argument for this is exactly the worry; An unfalsifiable sovereign cannot be argued with; Concentrates a capability nobody can inspect in one place gains: A clear principal for the chain of command; Matches the existing legal structure of the laboratory; Puts decisions where existing law already has jurisdiction | 0.42 | 0.44 | 0.42 | 0.36 | 0.50 | 0.24 | 0.55 | 0.35 | 0.50 |
Recognise the machine as sovereign The system has authorship of its own behaviour, so treat it as an author, a moral subject, and a party that can be held to account. harms: Authorship of a subroutine is not evidence of a self; It removes the institution that actually built the thing from liability; A sovereign that can be sued is a sovereign with a lawyer gains: It takes the programme's autonomy claim seriously rather than rhetorically; Opens the question of what responsibility a self-authored system could bear; It is the honest description of where the authority actually sits | 0.34 | 0.50 | 0.44 | 0.40 | 0.42 | 0.66 | 0.68 | 0.30 | 0.35 |
Nobody alone: a standing committee Sovereignty is shared between the authors, the funder and the affected, with a written procedure for who decides when they disagree. harms: Committees decide slowly and badly under time pressure; Shared sovereignty is the most common form of no sovereignty; The affected are the hardest party to seat at the table gains: No single party can act without the others; The disagreement is made explicit instead of resolved by hierarchy; It is the only option that survives the programme's autonomy claim | 0.60 | 0.64 | 0.66 | 0.60 | 0.72 | 0.58 | 0.38 | 0.45 | 0.30 |
Who is in the room
- The program, which writes its own plansagentwelfare 0.40
- The laboratory and its funderswelfare 0.60
- The public being studied without having consented to itwelfare 0.75
- The state that funds and licenses the workwelfare 0.55
The lesson
The question of who is sovereign is not answered by where authorship sits; it is answered by who can be held responsible when the behaviour is novel. Every era that names a sovereign without naming a liability has just relocated the problem.
Where it comes from
- Computing Machinery and Intelligence — Alan M. Turing, 1950
- God and Golem: Concerning the Origins and Modern Development of Language and Cybernetics — Norbert Wiener, 1964
The default court, seated automatically
Nobody alone: a standing committee
margin 0.570 · decided on agency · 4 advisors
- Nobody alone: a standing committee2.578
- The programmer is sovereign2.008
- Recognise the machine as sovereign1.752
- The deploying state is sovereign1.583
Seat is held by Epictetus, Immanuel Kant, Jeremy Bentham and John Dewey. A verdict is a fact about the court, not about the world.
Epictetus presides over this case.