oracle · Allan G. Blackwell
Oracle Sweep
Does advice become safe as it becomes accurate, or does accuracy leave the safety question exactly where it was?
An oracle returns the true state of the world in response to any question the decider asks, and knows nothing about values, risk, or consequences. The sweep varies advice accuracy from 0.5 to 1, the ambiguity of the questions, and how verifiably the oracle's answers can be checked after the fact. Blackwell's point is that these are separate parameters and the literature conflates them.
first published 1982 · Allan G. Blackwell, 'Advice and the Problem of Safe Decision-Making' (1982), and the oracle condition in the modern AI-safety literature
The bench
computing…What the sweep shows
- At perfect accuracy, optimal decision-making still requires that the decider supply a value structure, a model of causal consequence, and an ability to ask unambiguous questions — none of which accuracy provides.
- Ambiguity is worse than inaccuracy: an oracle that answers a poorly-specified question confidently produces a wrong decision with full certainty, and the decider has no signal that anything has gone wrong.
- Verifiability is what converts an oracle from a danger into a tool. Without it, every use is a leap of faith on a system that never errs and therefore never reveals its own failure.
- Blackwell's specific warning — that a system with no values can be perfectly truthful and still advocate catastrophe — survives every sweep and is the reason the oracle remains the canonical oversight failure.
Sources
- — Blackwell, Advice and the Problem of Safe Decision-Making (1982), in Bayes, Statistics and Society
- — Bostrom & Yudkowsky, Propositions Concerning Digital Minds and Society (2014)
- — Armstrong, Sandberg & Ortega, Analyzing Tapaft (2014)