Catastrophic forgetting
The loss of previously learned capability when a model is trained on new data.
What it commits you to
Fine-tuning a model on a narrow distribution reliably destroys performance elsewhere, which is the reason preference optimisation is so hard to scope: every constraint you add is trained against and something is lost. It also means that a safety fine-tune is not additive, and that removing it restores whatever was there before — including whatever was there before that was also unsafe.
Which realms lean on this
The vocabulary of measurement, and therefore of the disagreements measurement cannot settle.
technical · 1 realm — The Longview Foundry.
See also
Supervised fine-tuning
technicalContinued training on labelled examples of the behaviour you want.
RLHF
technicalReinforcement learning from human feedback: optimising a policy against learned preferences.
Reward hacking
alignmentGaining high reward without doing the thing the reward was meant to represent.
Policy
technicalThe mapping from states to actions that a trained system actually follows.