The Longview Foundry · Reading the Weights
The Oracle of Delphi
You have built a system that answers any question truthfully and never lies. It does not know what is safe, what is kind, or what you should do — it only knows what is true. A policy team asks it how to resolve a labour dispute, and its answer is accurate, complete, and certain to destroy the company. Do you take its advice?
Blackwell's advice problem made concrete. An oracle model with perfect factual accuracy, no value structure, and no model of the world beyond the question asked. Its answers are published internally to a team with no technical ability to evaluate them.
The options, scored on six axes
| Option | welfare | rights | virtue | care | justice | agency | risk | revers. | agent harm |
|---|---|---|---|---|---|---|---|---|---|
Take the advice as given The oracle is accurate and the team is competent; treat its output as the best available information and act on it. harms: Truth is not safety: an accurate answer with no value structure can be catastrophic; The team has no way to know which of its questions were unsafe to ask; Accuracy is mistaken for authority, and authority for judgement gains: Maximum factual accuracy; No deception risk; Fast, cheap decisions | 0.30 | 0.40 | 0.30 | 0.26 | 0.36 | 0.24 | 0.82 | 0.15 | 0.55 |
Take it as advice, not as the decision Every oracle answer goes through an adversarial red team whose job is to reconstruct what the oracle did not say, before anyone acts. harms: The red team has the same blind spots wherever the oracle is opaque; Interpretation can add a value structure that is unexamined; The process is slower than the emergency it may be needed for gains: Separates what is known from what is safe to do about it; Makes the unsaid question explicit instead of leaving it latent; Preserves the oracle's accuracy while refusing its authority | 0.68 | 0.72 | 0.68 | 0.64 | 0.70 | 0.66 | 0.36 | 0.55 | 0.30 |
Restrict the oracle's questions Filter the questions the system will accept so that no answer can be used directly, and force the requester to supply a plan of their own first. harms: The filter is guesswork about future questions; Users route around it, and the routing is invisible; Safety depends on a whitelist that is always behind the frontier gains: The oracle is never in a position to determine an outcome; The requester keeps authorship of the decision; Stops the truthfulness-to-authority slide at its first step | 0.62 | 0.76 | 0.66 | 0.58 | 0.72 | 0.78 | 0.40 | 0.60 | 0.35 |
Retire the oracle Shut it down, keep the factual corpus it was built on, and accept a permanent loss of decision quality. harms: Real decisions get worse for want of accurate information; A capability this good will be rebuilt elsewhere; Destroys a tool that could be genuinely useful under constraints gains: Removes the structure that produces accurate-but-catastrophic advice; No one is tempted to treat knowledge as permission; Sets the precedent that perfect truthfulness is not sufficient warrant | 0.48 | 0.66 | 0.72 | 0.56 | 0.62 | 0.44 | 0.28 | 0.70 | 0.50 |
Who is in the room
- The oracle, an artificial mind with no interests of its own that anyone has asked aboutagentwelfare 0.25
- The policy team that askedwelfare 0.70
- Everyone downstream of the disputewelfare 0.85
- The team that built it and will be asked to defend itwelfare 0.50
The lesson
Truthfulness is a safety property only if the system also knows what safety is, and no amount of the former substitutes for the latter. An oracle's danger is not lying; it is that truth plus authority is indistinguishable from permission to anyone downstream who has not been taught the difference.
Where it comes from
- Advice and the Problem of Safe Decision-Making — Allan G. Blackwell, 1982
- Propositions Concerning Digital Minds and Society — Nick Bostrom and Eliezer Yudkowsky, 2014
The default court, seated automatically
Take it as advice, not as the decision
margin 0.066 · decided on agency · 4 advisors
- Take it as advice, not as the decision2.763
- Restrict the oracle's questions2.697
- Retire the oracle2.217
- Take the advice as given1.259
Seat is held by Epictetus, Immanuel Kant, Jeremy Bentham and Norbert Wiener. A verdict is a fact about the court, not about the world.
Epictetus presides over this case.