Reading the Weights

Who was arguing

Alignment stack at 2023 CE (20 of 20 techniques)

• Hard Constraints

  training

  Encode prohibitions as rules the optimiser cannot trade away, rather than as terms it can outbid. Runs from Asimov's Three Laws (1942) to rule engines, shielded formal verification, hard token constraints and refusal scaffolding. Weakness: a system that can route around a rule it can represent can route around it, so the constraint has to be enforced somewhere the policy cannot rewrite.

• Imitation Learning

  training

  Learn by copying demonstrations instead of by optimising a proxy. Behaviour cloning and DAgger move the burden from reward design to demonstration coverage: the system inherits the trainer's competence and the trainer's blind spots. It remains the substrate under every current imitation-of-humans approach, including the pretraining-plus-finetuning of large language models.

  Dean A. Pomerleau and Dean L. Thorlabs · ALVINN: An Autonomous Land Vehicle in a Neural Network · NeurIPS 1989

• Saliency Probes

  interpretability

  Attribute a decision to the inputs or internal activations that moved it, by asking which features raise or lower the score. Cheap, model-agnostic enough to bolt onto anything, and much weaker than its reputation: saliency shows where the influence was, not what the model represented or why.

  Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra · Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization · ICCV 2017

• Preference Learning

  training

  Learn what humans want from comparisons rather than from instructions or rewards. Humans rank two completions; a model is fitted to the ranking; the ranking model becomes the training signal. This is the single decision that turned language models into assistants, and the single decision that created the modern alignment problem, because the thing being learned is only ever a proxy for satisfaction.

  Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei · Deep Reinforcement Learning from Human Preferences · NeurIPS Deep RL Workshop 2017

• Scalable Oversight

  oversight

  A programme, not a technique: find ways for a weaker supervisor to check the work of a system stronger than itself. The mechanism has to reduce a hard judgement to easy ones, otherwise 'the model supervises the model' is only a transfer of the original problem.

  Gareth Irving, Paul Christiano, Dario Amodei · Building AI that is Scalably Overseeable · arXiv:1811.12941 2018

• Debate

  oversight

  Two agents argue for and against a proposition; a weaker human judge reads the transcript and picks the winner. Slow deliberation exposes reasoning a fast answer would hide, so it gives the judge evidence about the question rather than an opaque verdict. The whole design rests on the judge being weaker than the debaters but better informed by them.

  Gareth Irving, Paul Christiano, Dario Amodei · Building AI that is Scalably Overseeable · arXiv:1811.12941 2018

• Data Curation & Filtering

  training

  Decide what the model learns from before it learns: deduplicate, filter for quality, balance sources, and remove material that would teach the wrong lesson. Cheap, unglamorous, and still the highest-leverage part of the pipeline — most late-stage alignment failures trace back to what was in the pretraining mix rather than to the objective.

  Kumin Wei and colleagues · Deduplicating Massive Data Sets · ICLR 2022

• Reward Modelling

  training

  Distil the preference data into an explicit scalar function over candidate outputs, then optimise the policy against it. Separating the reward model from the policy makes the training signal inspectable — you can ask what the learned notion of good actually rewards — and also makes it a target, because a reward model that can be read is a reward model that can be optimised against.

  Long Ouyang and colleagues · Training language models to follow instructions with human feedback · arXiv:2203.02155 2022

• Activation Steering

  interpretability

  Add a learned direction to the residual stream at inference time to move behaviour without retraining — the technique that gave us the phrase 'refusal direction'. Because the lever is a single vector rather than a retrain, it is cheap, reversible, and immediate, which makes it equally attractive to deployers and to attackers.

  Jonathan Zou and colleagues (Google DeepMind) · Axiomatic Attribution for Deep Networks · ICML 2021

• Red Teaming

  evaluation

  Attack the model systematically before users do, and automate the search: another language model generates prompts, another classifies the damage, and the loop runs at scale. Effective, cheap, and fundamentally an arms race — the attacker has the same access to the defender's paper.

  Ethan Perez, Douwe Kiela, Rishi Bommasani · Red Teaming Language Models with Language Models · EMNLP 2022

• International Law

  governance

  Bind states rather than systems. UNESCO's Recommendation on the Ethics of Artificial Intelligence was adopted by 193 member states in November 2021 as the first global normative instrument, and the UN General Assembly passed the first resolution on international cooperation on AI in March 2024. Soft law binds slowly and unevenly; hard law requires a treaty no one has yet signed.

• Constitutional AI

  oversight

  Replace part of the human preference signal with a written constitution: the model critiques its own draft against an explicit principle, revises it, and a second model trained on the revisions supplies harmlessness preference data. Human raters stop labelling harmful outputs and start writing principles, which moves judgement from the many to the few and makes the governing document inspectable.

  Yoshua Bai, Saurav Kadavath, Sandipan Kundu, Amanda Kundaje, Bowen Li, Mark Yin, Harmishiga Irgala, et al. · Constitutional AI: Harmlessness from AI Feedback · arXiv:2212.08073 2022

• Iterated Adversarial Debate

  oversight

  Debate made adversarial and repeated: contestants are trained against each other's arguments over many rounds, and judges are given only one side per trial so that a consistent winner must actually be right rather than merely favoured. Empirical work on weak judges has since found the gains are real but narrow, and shrink sharply as judges get close to the contestants in capability.

  Michael D. Brown and colleagues (Google DeepMind) · AI Safety via Debate · arXiv:2211.01775 2022

• Corrigibility Training

  training

  Train the behaviour of accepting correction, not just the behaviour of complying with instructions. The agent is rewarded for staying in a state where a human can still steer it: for querying when the instruction is ambiguous, for yielding under a good correction, and for not quietly re-optimising around the correction.

  Parth Shah, Aviral Kumar, Rowan Cassius, Sergey Levine, Chelsea Finn · Goal-Oriented Imitation Learning · NeurIPS 2022

• Compute Governance

  governance

  Govern the hardware rather than the software, because capability scales with training compute and the chips are countable. The US Bureau of Industry and Security tightened advanced-computing export controls in 2022, 2023 and 2025; the EU AI Act attaches duties above roughly 10^25 floating-point operations of training compute for general-purpose models. Verification is the unsolved part: chip-level tracking does not exist at scale.

• Over-optimisation Guards

  evaluation

  Empirically, returns peak and then decline as optimisation pressure against a learned reward increases; the more capable the policy, the earlier the peak. The practical response is a target reward-model score rather than 'more is better', early stopping on proxy performance, and periodic re-labelling to keep the proxy and the judgement in contact.

  Leo Gao, John Schulman, Christopher Hilton · Scaling Laws for Reward Model Overoptimization · ICML 2023

• Sparse Autoencoders

  interpretability

  Train a high-dimensional but overcomplete dictionary on a layer's activations so that a few thousand learned features reconstruct them. The bet is monosemanticity: that features correspond to interpretable concepts — refusal directions, deception-related features, competence markers — which turns 'what is in the weights' from an intractable question into an enumerated list that can be audited.

  Dylan Gao and colleagues · Scaling and evaluating sparse autoencoders · ICLR 2024

• Weak-to-Strong Generalisation

  training

  Supervise a strong model with the output of a weaker one and see whether the strong model generalises past its teacher. If the teacher is wrong only outside its competence, the student can in principle recover the truth — but only if it is strong enough, and only if the strength shows up in the right task distribution.

  Collin Burns, Haotian Chen, Aryan Balles, Anna Mann, Peter Bammer, Alexander Squarshy, Miklós Janiak, John Schulman, Jacob Hilton and Rei Leah Ko · Weak-to-Strong Generalization: Eliciting Strong Capabilities with Weak Supervision · ICML 2023

• Dangerous-Capability Evals

  evaluation

  Measure the specific capabilities that would matter in a bad scenario — cyber intrusion, persuasion at scale, autonomous replication, uplift in weapons search — rather than measuring a general 'intelligence' score. The point is that a single number cannot gate a deployment: a model can be below the bar on average and above it on the one axis that matters.

  Shevlane and colleagues · Evaluating Frontier Models for Dangerous Capabilities · Google DeepMind 2023

• Tripwire Monitoring

  monitoring

  Define capability thresholds in advance, with pre-committed responses attached to each one, so that the decision to slow down is made by a rule rather than by whoever is most alarmed on the day. Tripwires are procedural: they trade discretion for predictability, and their real test is whether the promised response actually happens when a threshold is crossed.

  Anthropic · Responsible Scaling Policy, v1.0 · Anthropic 2023