Weak-to-strong generalisation
A strong model trained on a weak supervisor's output generalising beyond its teacher.
What it commits you to
Weak-to-strong generalisation is the empirical finding that a strong model supervised by a weaker one often does better than the teacher, presumably because capability carries information the supervisor lacks. It matters for alignment because it suggests human and AI supervision can substitute for each other. It is also a warning: on the tasks where the teacher is confidently wrong, the student is reliably wrong too.
Which realms lean on this
Every realm has to keep a system honest. These are the terms that say out loud what the other kinds keep implicit.
alignment · 5 realms — The Benthamite Foundry, The Existential Wilds, The Longview Foundry, The Stoic Ramparts, The Ubuntu Weave.
See also
Scalable oversight
alignmentSupervision methods that let a weaker checker check something stronger than itself.
Model organism
alignmentA small system studied closely enough that its failures teach you something general.
Fallibilism
philosophyThe view that our beliefs are revisable and that confidence is never final.
RLHF
technicalReinforcement learning from human feedback: optimising a policy against learned preferences.