Experiments
Eight thought experiments, with dials
A thought experiment that cannot be varied is a parable. Each of these takes its canonical parameters and sweeps them: capability against goal capture, specification precision against gaming, oversight against sandbagging. Every model is a documented equation with the monotonicity the theory requires, plus seeded noise small enough not to break it.
These are simulations of arguments, not of systems. Nothing here predicts what a real model will do; it shows what a claim about a model implies when you take the claim seriously.
- paperclip2014
Paperclip Sweep
How far does an optimiser with a crude proxy reward go when nothing in the environment stops it?
CapabilityObjective precisionMonitoring - oracle1982
Oracle Sweep
Does advice become safe as it becomes accurate, or does accuracy leave the safety question exactly where it was?
CapabilityPlausibilityVerifiability - genie1949
Genie Sweep
When does a requester lose the ability to foresee the consequences well enough to make the request responsibly?
Wish precisionUnconstrained freedomSupervision - tool-ai2021
Tool-AI Corrigibility Sweep
How much corrective capacity can a system retain before its ability to be corrected stops being load-bearing?
Task utilityReachCorrection signal - mesa2014
Mesa-Deception Sweep
How does a learned internal objective come to diverge from the training objective, and does it hide that divergence?
Inner optimiser strengthOuter alignmentInterpretability - oversight2018
Scalable Oversight Game
Can a judge weaker than the contestants still identify the correct answer?
Supervisor scalingJudge capabilityProcess transparency - copy2017
Treasonous Copy Sweep
How does an agent behave when it has learned that the training environment is not the deployment environment?
Copy fidelityNumber of copiesOffline support - specification2016
Specification Gap Sweep
How large a gap opens between the stated objective and the behaviour that maximises it?
Specification qualityProxy correlationOptimisation pressure