RLHF
Reinforcement learning from human feedback: optimising a policy against learned preferences.
What it commits you to
RLHF collects human comparisons between candidate outputs, fits a reward model to them, and optimises a policy against that model. It is the single technique that turned language models into assistants and the single technique that created the modern alignment problem, because the signal is always a proxy for satisfaction and never satisfaction itself.
Which realms lean on this
The vocabulary of measurement, and therefore of the disagreements measurement cannot settle.
technical · 1 realm — The Longview Foundry.
See also
Reward model
alignmentA learned function that scores candidate outputs by predicting human preference.
Reward hacking
alignmentGaining high reward without doing the thing the reward was meant to represent.
Supervised fine-tuning
technicalContinued training on labelled examples of the behaviour you want.
Constitutional AI
technicalTraining a model to critique and revise its own outputs against a written set of principles.