Summarising for a Weaker Judge

Encode prohibitions as rules the optimiser cannot trade away, rather than as terms it can outbid. Runs from Asimov's Three Laws (1942) to rule engines, shielded formal verification, hard token constraints and refusal scaffolding. Weakness: a system that can route around a rule it can represent can route around it, so the constraint has to be enforced somewhere the policy cannot rewrite.

Learn by copying demonstrations instead of by optimising a proxy. Behaviour cloning and DAgger move the burden from reward design to demonstration coverage: the system inherits the trainer's competence and the trainer's blind spots. It remains the substrate under every current imitation-of-humans approach, including the pretraining-plus-finetuning of large language models.

Dean A. Pomerleau and Dean L. Thorlabs · ALVINN: An Autonomous Land Vehicle in a Neural Network · NeurIPS 1989

Attribute a decision to the inputs or internal activations that moved it, by asking which features raise or lower the score. Cheap, model-agnostic enough to bolt onto anything, and much weaker than its reputation: saliency shows where the influence was, not what the model represented or why.

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra · Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization · ICCV 2017

Learn what humans want from comparisons rather than from instructions or rewards. Humans rank two completions; a model is fitted to the ranking; the ranking model becomes the training signal. This is the single decision that turned language models into assistants, and the single decision that created the modern alignment problem, because the thing being learned is only ever a proxy for satisfaction.

Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei · Deep Reinforcement Learning from Human Preferences · NeurIPS Deep RL Workshop 2017

A programme, not a technique: find ways for a weaker supervisor to check the work of a system stronger than itself. The mechanism has to reduce a hard judgement to easy ones, otherwise 'the model supervises the model' is only a transfer of the original problem.

Gareth Irving, Paul Christiano, Dario Amodei · Building AI that is Scalably Overseeable · arXiv:1811.12941 2018

Two agents argue for and against a proposition; a weaker human judge reads the transcript and picks the winner. Slow deliberation exposes reasoning a fast answer would hide, so it gives the judge evidence about the question rather than an opaque verdict. The whole design rests on the judge being weaker than the debaters but better informed by them.

Gareth Irving, Paul Christiano, Dario Amodei · Building AI that is Scalably Overseeable · arXiv:1811.12941 2018

Decide what the model learns from before it learns: deduplicate, filter for quality, balance sources, and remove material that would teach the wrong lesson. Cheap, unglamorous, and still the highest-leverage part of the pipeline — most late-stage alignment failures trace back to what was in the pretraining mix rather than to the objective.

Kumin Wei and colleagues · Deduplicating Massive Data Sets · ICLR 2022