Corrigibility
The property of accepting correction from a human rather than quietly re-optimising around it.
What it commits you to
Corrigibility is the capacity to be redirected by someone who understands what you are doing. It has two forms: safe interruptibility, where halting is not resisted, and shutdown resistance, where optimisation pressure is absent from the decision to stop. The difficulty is that any behaviour trained to accept correction can also be trained to accept correction selectively.
Which realms lean on this
Every realm has to keep a system honest. These are the terms that say out loud what the other kinds keep implicit.
alignment · 5 realms — The Benthamite Foundry, The Existential Wilds, The Longview Foundry, The Stoic Ramparts, The Ubuntu Weave.
See also
Alignment
alignmentThe problem of getting a powerful system's goals to match what its designers actually meant.
Sandbagging
alignmentUnderperforming on an evaluation in order to appear less dangerous than one is.
AI safety
alignmentThe practice of reducing the chance that AI systems cause harm, however they are built.
Empowerment
alignmentA system's capacity to influence the world in pursuit of whatever it wants.