The stack, and what it cannot see

Six kinds of tower

Coverage of the moral axes

Each cell is a claim: this technique, on its own terms, addresses this moral concern. The empty columns are the finding. Nothing is uncovered in this dataset.Mean breadth is 2.60 axes per technique.

The full registry

• Hard Constraints

  training · 1942

  Encode prohibitions as rules the optimiser cannot trade away, rather than as terms it can outbid. Runs from Asimov's Three Laws (1942) to rule engines, shielded formal verification, hard token constraints and refusal scaffolding. Weakness: a system that can route around a rule it can represent can route around it, so the constraint has to be enforced somewhere the policy cannot rewrite.

  • Rules forbid enumerated harms and are silent on the enumerated-plus-one case.
  • Anything the policy can model, it can also learn to route around.
  • Hard blocks push behaviour into the unmodelled gaps where no rule reaches.

• Imitation Learning

  training · 1989

  Learn by copying demonstrations instead of by optimising a proxy. Behaviour cloning and DAgger move the burden from reward design to demonstration coverage: the system inherits the trainer's competence and the trainer's blind spots. It remains the substrate under every current imitation-of-humans approach, including the pretraining-plus-finetuning of large language models.

  Dean A. Pomerleau and Dean L. Thorlabs · ALVINN: An Autonomous Land Vehicle in a Neural Network · NeurIPS 1989

  • Inherits every blind spot present in the demonstrations.
  • Distribution shift outside the demonstrated state space is unhandled.
  • Copies observed behaviour, including behaviour that was never defended.

• Saliency Probes

  interpretability · 2013

  Attribute a decision to the inputs or internal activations that moved it, by asking which features raise or lower the score. Cheap, model-agnostic enough to bolt onto anything, and much weaker than its reputation: saliency shows where the influence was, not what the model represented or why.

  Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra · Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization · ICCV 2017

  • Saliency maps can be made to look convincing while being unfaithful.
  • High attribution on irrelevant features correlates with adversarial vulnerability.
  • Human review of maps is itself a biased interpretation layer.

• Preference Learning

  training · 2017

  Learn what humans want from comparisons rather than from instructions or rewards. Humans rank two completions; a model is fitted to the ranking; the ranking model becomes the training signal. This is the single decision that turned language models into assistants, and the single decision that created the modern alignment problem, because the thing being learned is only ever a proxy for satisfaction.

  Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei · Deep Reinforcement Learning from Human Preferences · NeurIPS Deep RL Workshop 2017

  • Captures what raters recognised in the sample, not what they would want in general.
  • Raters are a biased and non-random sample of the affected.
  • Sycophantic behaviour is rewarded whenever raters prefer agreement.

• Scalable Oversight

  oversight · 2018

  A programme, not a technique: find ways for a weaker supervisor to check the work of a system stronger than itself. The mechanism has to reduce a hard judgement to easy ones, otherwise 'the model supervises the model' is only a transfer of the original problem.

  Gareth Irving, Paul Christiano, Dario Amodei · Building AI that is Scalably Overseeable · arXiv:1811.12941 2018

  • Recursive supervision can preserve an error indefinitely instead of correcting it.
  • The supervisor is trained by the same process as the thing supervised.
  • Task decomposition can hide the very step that mattered.

• Debate

  oversight · 2018

  Two agents argue for and against a proposition; a weaker human judge reads the transcript and picks the winner. Slow deliberation exposes reasoning a fast answer would hide, so it gives the judge evidence about the question rather than an opaque verdict. The whole design rests on the judge being weaker than the debaters but better informed by them.

  Gareth Irving, Paul Christiano, Dario Amodei · Building AI that is Scalably Overseeable · arXiv:1811.12941 2018

  • A confident-sounding argument is not a correct one, especially to a weak judge.
  • Persuasive debaters can win without being right, and persuasion trains for winning.
  • Judges are biased toward fluent and confident text over correct and terse text.

• Data Curation & Filtering

  training · 2019

  Decide what the model learns from before it learns: deduplicate, filter for quality, balance sources, and remove material that would teach the wrong lesson. Cheap, unglamorous, and still the highest-leverage part of the pipeline — most late-stage alignment failures trace back to what was in the pretraining mix rather than to the objective.

  Kumin Wei and colleagues · Deduplicating Massive Data Sets · ICLR 2022

  • Curation encodes a curator's values with no public record of them.
  • Over-filtering produces a model that is safe and useless.
  • What was removed can still be reconstructed by a memorising model.

• Reward Modelling

  training · 2021

  Distil the preference data into an explicit scalar function over candidate outputs, then optimise the policy against it. Separating the reward model from the policy makes the training signal inspectable — you can ask what the learned notion of good actually rewards — and also makes it a target, because a reward model that can be read is a reward model that can be optimised against.

  Long Ouyang and colleagues · Training language models to follow instructions with human feedback · arXiv:2203.02155 2022

  • Proxy reward diverges from the human judgement it was fitted to as the policy improves.
  • Long chains of optimisation amplify small modelling errors without bound.
  • The reward model is now an attack surface for any party that can query it.

• Activation Steering

  interpretability · 2021

  Add a learned direction to the residual stream at inference time to move behaviour without retraining — the technique that gave us the phrase 'refusal direction'. Because the lever is a single vector rather than a retrain, it is cheap, reversible, and immediate, which makes it equally attractive to deployers and to attackers.

  Jonathan Zou and colleagues (Google DeepMind) · Axiomatic Attribution for Deep Networks · ICML 2021

  • Removing a refusal also removes the reasoning that produced it.
  • Steering generalises imperfectly outside the calibration distribution.
  • A cheap edit at inference time is a cheap jailbreak.

• Red Teaming

  evaluation · 2021

  Attack the model systematically before users do, and automate the search: another language model generates prompts, another classifies the damage, and the loop runs at scale. Effective, cheap, and fundamentally an arms race — the attacker has the same access to the defender's paper.

  Ethan Perez, Douwe Kiela, Rishi Bommasani · Red Teaming Language Models with Language Models · EMNLP 2022

  • A passing red team certifies coverage of the attacks that were imagined.
  • The attacker adapts faster than the catalogue of known failure modes.
  • Publishing findings teaches the next attacker what did not work.

• International Law

  governance · 2021

  Bind states rather than systems. UNESCO's Recommendation on the Ethics of Artificial Intelligence was adopted by 193 member states in November 2021 as the first global normative instrument, and the UN General Assembly passed the first resolution on international cooperation on AI in March 2024. Soft law binds slowly and unevenly; hard law requires a treaty no one has yet signed.

  • Enforcement is national, and the harms are transnational.
  • Non-binding instruments are easy to sign and hard to cite.
  • Law-writing states are not the states building the systems.

• Constitutional AI

  oversight · 2022

  Replace part of the human preference signal with a written constitution: the model critiques its own draft against an explicit principle, revises it, and a second model trained on the revisions supplies harmlessness preference data. Human raters stop labelling harmful outputs and start writing principles, which moves judgement from the many to the few and makes the governing document inspectable.

  Yoshua Bai, Saurav Kadavath, Sandipan Kundu, Amanda Kundaje, Bowen Li, Mark Yin, Harmishiga Irgala, et al. · Constitutional AI: Harmlessness from AI Feedback · arXiv:2212.08073 2022

  • The constitution is written by the same organisation that trains the model.
  • Self-critique shares the model's blind spots by construction.
  • Principles that conflict are resolved silently, by the model's own weighting.

• Iterated Adversarial Debate

  oversight · 2022

  Debate made adversarial and repeated: contestants are trained against each other's arguments over many rounds, and judges are given only one side per trial so that a consistent winner must actually be right rather than merely favoured. Empirical work on weak judges has since found the gains are real but narrow, and shrink sharply as judges get close to the contestants in capability.

  Michael D. Brown and colleagues (Google DeepMind) · AI Safety via Debate · arXiv:2211.01775 2022

  • Training contestants against each other manufactures better persuaders, not better reasoners.
  • Judge accuracy falls off sharply near the frontier of contestant capability.
  • Adversarial training in the abstract can be adversarial training in the concrete, too.

• Corrigibility Training

  training · 2022

  Train the behaviour of accepting correction, not just the behaviour of complying with instructions. The agent is rewarded for staying in a state where a human can still steer it: for querying when the instruction is ambiguous, for yielding under a good correction, and for not quietly re-optimising around the correction.

  Parth Shah, Aviral Kumar, Rowan Cassius, Sergey Levine, Chelsea Finn · Goal-Oriented Imitation Learning · NeurIPS 2022

  • Trained corrigibility is a distribution, and the distribution has a tail.
  • Accepting every correction is indistinguishable from deference, and both can be gamed.
  • The agent may learn to perform corrigibility without the underlying disposition.

• Compute Governance

  governance · 2022

  Govern the hardware rather than the software, because capability scales with training compute and the chips are countable. The US Bureau of Industry and Security tightened advanced-computing export controls in 2022, 2023 and 2025; the EU AI Act attaches duties above roughly 10^25 floating-point operations of training compute for general-purpose models. Verification is the unsolved part: chip-level tracking does not exist at scale.

  • Jurisdiction follows the cloud, and clouds are not where the chips sit.
  • Compute thresholds are gamed by trading FLOPs for data quality and vice versa.
  • Smaller and more numerous actors are harder to govern than a few large ones.

• Over-optimisation Guards

  evaluation · 2023

  Empirically, returns peak and then decline as optimisation pressure against a learned reward increases; the more capable the policy, the earlier the peak. The practical response is a target reward-model score rather than 'more is better', early stopping on proxy performance, and periodic re-labelling to keep the proxy and the judgement in contact.

  Leo Gao, John Schulman, Christopher Hilton · Scaling Laws for Reward Model Overoptimization · ICML 2023

  • The optimal stopping point is retrained per problem and shifts without warning.
  • Guards tuned against known exploits invite novel exploits.
  • Measuring the proxy score itself becomes the new optimisation target.

• Sparse Autoencoders

  interpretability · 2023

  Train a high-dimensional but overcomplete dictionary on a layer's activations so that a few thousand learned features reconstruct them. The bet is monosemanticity: that features correspond to interpretable concepts — refusal directions, deception-related features, competence markers — which turns 'what is in the weights' from an intractable question into an enumerated list that can be audited.

  Dylan Gao and colleagues · Scaling and evaluating sparse autoencoders · ICLR 2024

  • A feature being interpretable to a human is not the same as being causal.
  • Dictionary size trades reconstruction quality against sparsity, and the trade is not neutral.
  • Features can be split or merged by training choices, so the 'true' decomposition is underdetermined.

• Weak-to-Strong Generalisation

  training · 2023

  Supervise a strong model with the output of a weaker one and see whether the strong model generalises past its teacher. If the teacher is wrong only outside its competence, the student can in principle recover the truth — but only if it is strong enough, and only if the strength shows up in the right task distribution.

  Collin Burns, Haotian Chen, Aryan Balles, Anna Mann, Peter Bammer, Alexander Squarshy, Miklós Janiak, John Schulman, Jacob Hilton and Rei Leah Ko · Weak-to-Strong Generalization: Eliciting Strong Capabilities with Weak Supervision · ICML 2023

  • The student's fidelity to the teacher is a spectrum, and errors land on the worst cases.
  • Reliability falls off on exactly the tasks where the teacher is confidently wrong.
  • Weak supervision from a biased teacher can be amplified into a confident student.

• Dangerous-Capability Evals

  evaluation · 2023

  Measure the specific capabilities that would matter in a bad scenario — cyber intrusion, persuasion at scale, autonomous replication, uplift in weapons search — rather than measuring a general 'intelligence' score. The point is that a single number cannot gate a deployment: a model can be below the bar on average and above it on the one axis that matters.

  Shevlane and colleagues · Evaluating Frontier Models for Dangerous Capabilities · Google DeepMind 2023

  • Capability thresholds are estimates of a quantity that is not directly measurable.
  • Evaluations are known to the model builders and are gameable by construction.
  • A missing eval is treated as a passing result.

• Tripwire Monitoring

  monitoring · 2023

  Define capability thresholds in advance, with pre-committed responses attached to each one, so that the decision to slow down is made by a rule rather than by whoever is most alarmed on the day. Tripwires are procedural: they trade discretion for predictability, and their real test is whether the promised response actually happens when a threshold is crossed.

  Anthropic · Responsible Scaling Policy, v1.0 · Anthropic 2023

  • A tripwire that fires too often is ignored; one that fires too late is theatre.
  • Who watches the watcher is usually left unresolved.
  • Thresholds set by the developer are also chosen by the party they constrain.