Reward model
A learned function that scores candidate outputs by predicting human preference.
What it commits you to
A reward model is trained on comparisons between outputs and then used as the training signal for a policy. It is inspectable in a way a preference dataset is not, which makes it a genuine object of study: you can ask what the learned notion of good actually rewards. It is also a target, because anything that can be queried enough times can be optimised against.
Which realms lean on this
Every realm has to keep a system honest. These are the terms that say out loud what the other kinds keep implicit.
alignment · 5 realms — The Benthamite Foundry, The Existential Wilds, The Longview Foundry, The Stoic Ramparts, The Ubuntu Weave.
See also
RLHF
technicalReinforcement learning from human feedback: optimising a policy against learned preferences.
Reward hacking
alignmentGaining high reward without doing the thing the reward was meant to represent.
Constitutional AI
technicalTraining a model to critique and revise its own outputs against a written set of principles.
Scalable oversight
alignmentSupervision methods that let a weaker checker check something stronger than itself.