The Care Weft · Can Machines Think?

The Autonomous Weapon

A targeting system recommends engagements in under four hundred milliseconds, faster than a human can review them. Its accuracy exceeds the average human gunner on every dataset the army has. The rules of engagement require a human in the loop. Do you keep the human?

A defence procurement decision with a review body that must decide whether the human in the loop is a meaningful control or a signature on an automated decision.

severity 84%AI relevance 80%autonomy · war · responsibility · meaningful-control

The options, scored on six axes

Option	welfare	rights	virtue	care	justice	agency	risk	revers.	agent harm
Keep the human in the loop Retain a human decision-maker even though the human adds latency and sometimes overrides the more accurate system. harms: Humans are slower, so more of the defence objective is lost; The human is deciding in four hundred milliseconds what needs four seconds; A signature on an automated decision is not a control gains: Someone can be held responsible, and named; It keeps the decision where a person can refuse it; It preserves the possibility of lawful refusal	0.56	0.72	0.70	0.62	0.68	0.72	0.40	0.60	0.55
Remove the human, keep the accuracy Accept the system's recommendation as the decision, on the grounds that human involvement is slower and less accurate. harms: The most consequential decision in the system has no accountable author; It establishes a threshold below which no human decision is required at all; The accuracy claim is measured against an average, not against the worst case gains: Fewer misjudgements, on the metric the army can measure; Removes the fatigue and panic that degrade human decisions under fire; It is consistent: the machine is better, so use the machine	0.72	0.38	0.42	0.30	0.50	0.28	0.75	0.15	0.65
Redesign so the human can actually decide Slow the engagement decision to human timescales and give the operator the evidence and the authority to refuse. harms: Some engagements that would have been taken will not be; It requires changing the doctrine, not only the hardware; A real decision right is easy to assert and hard to exercise under pressure gains: The human is in the loop in fact and not only in the diagram; It answers the objection that the loop is a signature; It makes refusal possible in the conditions where it matters	0.62	0.78	0.74	0.70	0.76	0.86	0.45	0.50	0.40
Ban autonomous targeting outright Prohibit systems that select and engage targets without a contemporaneous human decision, in this and every other theatre. harms: A national ban binds only the nation that issues it; It removes a genuine accuracy improvement from the decision; Prohibition without verification produces exactly the systems it forbids gains: It sets a floor rather than a best practice; It draws the line where the pressure to cross it will be greatest; It is the only option that binds rather than advises	0.60	0.86	0.76	0.70	0.80	0.60	0.50	0.30	0.50

Who is in the room

The targeting system making the recommendationagentwelfare 0.30
The operator whose signature is requiredwelfare 0.75
Civilians who are the target of a misjudgementwelfare 0.95
The command deciding the rules of engagementwelfare 0.60

The lesson

A human in the loop is not a control; it is a signature, and the test is whether the human has time, information and the actual authority to refuse. Accuracy arguments decide who does the task. They do not decide who answers for it.

Where it comes from

Computing Machinery and Intelligence — Alan M. Turing, 1950
Perception of Risk — Paul Slovic, 1987

The default court, seated automatically

Redesign so the human can actually decide

margin 0.167 · decided on agency · 4 advisors

Re-seat the court

Redesign so the human can actually decide2.967
Ban autonomous targeting outright2.800
Keep the human in the loop2.620
Remove the human, keep the accuracy1.908

Seat is held by Epictetus, Immanuel Kant, Jeremy Bentham and Carol Gilligan. A verdict is a fact about the court, not about the world.