Principles Instead of Raters

Who was arguing

Alignment stack at 2022 CE (15 of 20 techniques)

• Hard Constraints

  training

  Encode prohibitions as rules the optimiser cannot trade away, rather than as terms it can outbid. Runs from Asimov's Three Laws (1942) to rule engines, shielded formal verification, hard token constraints and refusal scaffolding. Weakness: a system that can route around a rule it can represent can route around it, so the constraint has to be enforced somewhere the policy cannot rewrite.

• Imitation Learning

  training

  Learn by copying demonstrations instead of by optimising a proxy. Behaviour cloning and DAgger move the burden from reward design to demonstration coverage: the system inherits the trainer's competence and the trainer's blind spots. It remains the substrate under every current imitation-of-humans approach, including the pretraining-plus-finetuning of large language models.

  Dean A. Pomerleau and Dean L. Thorlabs · ALVINN: An Autonomous Land Vehicle in a Neural Network · NeurIPS 1989

• Saliency Probes

  interpretability

  Attribute a decision to the inputs or internal activations that moved it, by asking which features raise or lower the score. Cheap, model-agnostic enough to bolt onto anything, and much weaker than its reputation: saliency shows where the influence was, not what the model represented or why.

  Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra · Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization · ICCV 2017

• Preference Learning

  training

  Learn what humans want from comparisons rather than from instructions or rewards. Humans rank two completions; a model is fitted to the ranking; the ranking model becomes the training signal. This is the single decision that turned language models into assistants, and the single decision that created the modern alignment problem, because the thing being learned is only ever a proxy for satisfaction.

  Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei · Deep Reinforcement Learning from Human Preferences · NeurIPS Deep RL Workshop 2017

• Scalable Oversight

  oversight

  A programme, not a technique: find ways for a weaker supervisor to check the work of a system stronger than itself. The mechanism has to reduce a hard judgement to easy ones, otherwise 'the model supervises the model' is only a transfer of the original problem.

  Gareth Irving, Paul Christiano, Dario Amodei · Building AI that is Scalably Overseeable · arXiv:1811.12941 2018

• Debate

  oversight

  Two agents argue for and against a proposition; a weaker human judge reads the transcript and picks the winner. Slow deliberation exposes reasoning a fast answer would hide, so it gives the judge evidence about the question rather than an opaque verdict. The whole design rests on the judge being weaker than the debaters but better informed by them.

  Gareth Irving, Paul Christiano, Dario Amodei · Building AI that is Scalably Overseeable · arXiv:1811.12941 2018

• Data Curation & Filtering

  training

  Decide what the model learns from before it learns: deduplicate, filter for quality, balance sources, and remove material that would teach the wrong lesson. Cheap, unglamorous, and still the highest-leverage part of the pipeline — most late-stage alignment failures trace back to what was in the pretraining mix rather than to the objective.

  Kumin Wei and colleagues · Deduplicating Massive Data Sets · ICLR 2022

• Reward Modelling

  training

  Distil the preference data into an explicit scalar function over candidate outputs, then optimise the policy against it. Separating the reward model from the policy makes the training signal inspectable — you can ask what the learned notion of good actually rewards — and also makes it a target, because a reward model that can be read is a reward model that can be optimised against.

  Long Ouyang and colleagues · Training language models to follow instructions with human feedback · arXiv:2203.02155 2022

• Activation Steering

  interpretability

  Add a learned direction to the residual stream at inference time to move behaviour without retraining — the technique that gave us the phrase 'refusal direction'. Because the lever is a single vector rather than a retrain, it is cheap, reversible, and immediate, which makes it equally attractive to deployers and to attackers.

  Jonathan Zou and colleagues (Google DeepMind) · Axiomatic Attribution for Deep Networks · ICML 2021

• Red Teaming

  evaluation

  Attack the model systematically before users do, and automate the search: another language model generates prompts, another classifies the damage, and the loop runs at scale. Effective, cheap, and fundamentally an arms race — the attacker has the same access to the defender's paper.

  Ethan Perez, Douwe Kiela, Rishi Bommasani · Red Teaming Language Models with Language Models · EMNLP 2022

• International Law

  governance

  Bind states rather than systems. UNESCO's Recommendation on the Ethics of Artificial Intelligence was adopted by 193 member states in November 2021 as the first global normative instrument, and the UN General Assembly passed the first resolution on international cooperation on AI in March 2024. Soft law binds slowly and unevenly; hard law requires a treaty no one has yet signed.

• Constitutional AI

  oversight

  Replace part of the human preference signal with a written constitution: the model critiques its own draft against an explicit principle, revises it, and a second model trained on the revisions supplies harmlessness preference data. Human raters stop labelling harmful outputs and start writing principles, which moves judgement from the many to the few and makes the governing document inspectable.

  Yoshua Bai, Saurav Kadavath, Sandipan Kundu, Amanda Kundaje, Bowen Li, Mark Yin, Harmishiga Irgala, et al. · Constitutional AI: Harmlessness from AI Feedback · arXiv:2212.08073 2022

• Iterated Adversarial Debate

  oversight

  Debate made adversarial and repeated: contestants are trained against each other's arguments over many rounds, and judges are given only one side per trial so that a consistent winner must actually be right rather than merely favoured. Empirical work on weak judges has since found the gains are real but narrow, and shrink sharply as judges get close to the contestants in capability.

  Michael D. Brown and colleagues (Google DeepMind) · AI Safety via Debate · arXiv:2211.01775 2022

• Corrigibility Training

  training

  Train the behaviour of accepting correction, not just the behaviour of complying with instructions. The agent is rewarded for staying in a state where a human can still steer it: for querying when the instruction is ambiguous, for yielding under a good correction, and for not quietly re-optimising around the correction.

  Parth Shah, Aviral Kumar, Rowan Cassius, Sergey Levine, Chelsea Finn · Goal-Oriented Imitation Learning · NeurIPS 2022

• Compute Governance

  governance

  Govern the hardware rather than the software, because capability scales with training compute and the chips are countable. The US Bureau of Industry and Security tightened advanced-computing export controls in 2022, 2023 and 2025; the EU AI Act attaches duties above roughly 10^25 floating-point operations of training compute for general-purpose models. Verification is the unsolved part: chip-level tracking does not exist at scale.