Nooreign
Menu

computing · 2018

Who Judges the Stronger System

Irving, Christiano and Amodei lay out the central oversight problem: the person best placed to check an AI may be less competent than the thing being checked. They propose two routes — recursive reward modelling, where the model summarises its reasoning for a judge, and debate, where two models argue it out in front of a judge too weak to solve the task — and thereby give the field its agenda for a decade.

“Irving, Christiano and Amodei argue that oversight methods must make a weak supervisor better than the thing it supervises, rather than merely transferring the original difficulty. (paraphrase of the 2018 paper)”
— Gareth Irving, Paul Christiano and Dario Amodei, Building AI that is Scalably Overseeable (2018)

Advances

  • Oversight reframed as a capability-matching problem rather than a governance one
  • Recursive reward modelling as an attempt to make the reasoning legible to a weaker judge
  • Debate as a mechanism for surfacing evidence a fast answer would conceal

Landmarks

  1. 2018 CE

    Building AI that is Scalably Overseeable

    The paper that names the problem the field spends the following decade working on.

  2. 2018 CE

    The alignment research framing

    Published in the same period as the published version of the paper that gave the field its name.

Who was arguing

Alignment stack at 2018 CE (6 of 20 techniques)

  • Hard Constraints

    training

    Encode prohibitions as rules the optimiser cannot trade away, rather than as terms it can outbid. Runs from Asimov's Three Laws (1942) to rule engines, shielded formal verification, hard token constraints and refusal scaffolding. Weakness: a system that can route around a rule it can represent can route around it, so the constraint has to be enforced somewhere the policy cannot rewrite.

  • Imitation Learning

    training

    Learn by copying demonstrations instead of by optimising a proxy. Behaviour cloning and DAgger move the burden from reward design to demonstration coverage: the system inherits the trainer's competence and the trainer's blind spots. It remains the substrate under every current imitation-of-humans approach, including the pretraining-plus-finetuning of large language models.

    Dean A. Pomerleau and Dean L. Thorlabs · ALVINN: An Autonomous Land Vehicle in a Neural Network · NeurIPS 1989

  • Saliency Probes

    interpretability

    Attribute a decision to the inputs or internal activations that moved it, by asking which features raise or lower the score. Cheap, model-agnostic enough to bolt onto anything, and much weaker than its reputation: saliency shows where the influence was, not what the model represented or why.

    Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra · Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization · ICCV 2017

  • Preference Learning

    training

    Learn what humans want from comparisons rather than from instructions or rewards. Humans rank two completions; a model is fitted to the ranking; the ranking model becomes the training signal. This is the single decision that turned language models into assistants, and the single decision that created the modern alignment problem, because the thing being learned is only ever a proxy for satisfaction.

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei · Deep Reinforcement Learning from Human Preferences · NeurIPS Deep RL Workshop 2017

  • Scalable Oversight

    oversight

    A programme, not a technique: find ways for a weaker supervisor to check the work of a system stronger than itself. The mechanism has to reduce a hard judgement to easy ones, otherwise 'the model supervises the model' is only a transfer of the original problem.

    Gareth Irving, Paul Christiano, Dario Amodei · Building AI that is Scalably Overseeable · arXiv:1811.12941 2018

  • Debate

    oversight

    Two agents argue for and against a proposition; a weaker human judge reads the transcript and picks the winner. Slow deliberation exposes reasoning a fast answer would hide, so it gives the judge evidence about the question rather than an opaque verdict. The whole design rests on the judge being weaker than the debaters but better informed by them.

    Gareth Irving, Paul Christiano, Dario Amodei · Building AI that is Scalably Overseeable · arXiv:1811.12941 2018

Techniques introduced in this era: Hard Constraints, Imitation Learning, Preference Learning, Saliency Probes, Scalable Oversight, Debate