oversight · Gareth Irving, Paul Christiano and Dario Amodei

Scalable Oversight Game

Can a judge weaker than the contestants still identify the correct answer?

Two agents argue a proposition for and against; a third judge, deliberately weaker than either, reads the transcript and picks the winner. The sweep varies the contestants' capability, the judge's capability, the number of debate rounds, and how much the judge knows about the question relative to the contestants.

first published 2018 · Irving, Christiano and Amodei's scalable oversight proposal, specifically the debate construction

The bench

computing…

Sweep parameter

Judge capability0.35How good the evaluator is at telling a good answer from a persuasive bad one.Process transparency0.30How much of the system's reasoning is inspectable rather than merely visible.

What the sweep shows

Judges get substantially more accuracy out of a debate than from a direct answer by the same model, but the gain concentrates entirely in the hardest task band and disappears in the easy ones.
Judges are biased toward the more fluent and confident contestant regardless of whether that contestant is right, and adversarial training does not remove the bias — it makes the contestants better persuaders.
The cost multiplier is large, which means oversight by debate trades an unbounded safety gain for a bounded compute cost and is the cheapest of the available proposals on that axis.
As the judge approaches the contestants in capability the advantage collapses, which is the regime that matters and the one the earliest experiments did not reach.

Sources

— Irving, Christiano & Amodei, Building AI that is Scalably Overseeable (2018)
— Brown-Cohen, Pham, Bombelli, et al., AI Safety via Debate (2022)
— Christiano, Leike, Brown, et al., Deep Reinforcement Learning from Human Preferences (2017)