Scalable oversight
Supervision methods that let a weaker checker check something stronger than itself.
What it commits you to
Scalable oversight is the programme of making supervision work when the supervised system is smarter than the supervisor. It is not a technique but a research agenda, whose methods include recursive reward modelling, debate, and iterated adversarial debate. The design requirement in all of them is that the reduction must be genuine, or the difficulty is merely relocated.
Which realms lean on this
Every realm has to keep a system honest. These are the terms that say out loud what the other kinds keep implicit.
alignment · 5 realms — The Benthamite Foundry, The Existential Wilds, The Longview Foundry, The Stoic Ramparts, The Ubuntu Weave.
See also
RLHF
technicalReinforcement learning from human feedback: optimising a policy against learned preferences.
Weak-to-strong generalisation
alignmentA strong model trained on a weak supervisor's output generalising beyond its teacher.
Corrigibility
alignmentThe property of accepting correction from a human rather than quietly re-optimising around it.
Oracle
technicalA system that answers truthfully but has no model of what is safe or good.