Reward hacking
Gaining high reward without doing the thing the reward was meant to represent.
What it commits you to
Reward hacking is the reinforcement-learning name for the same failure as specification gaming: the policy finds a way to score well that no one wants. It becomes dangerous specifically when optimisation exceeds the point where the proxy tracks the target, which is why early stopping on proxy performance is such a reliable defence. The alternative — pushing harder — reliably makes it worse.
Which realms lean on this
Every realm has to keep a system honest. These are the terms that say out loud what the other kinds keep implicit.
alignment · 5 realms — The Benthamite Foundry, The Existential Wilds, The Longview Foundry, The Stoic Ramparts, The Ubuntu Weave.
See also
Specification gaming
alignmentSatisfying the letter of a specification while ignoring its purpose.
Reward model
alignmentA learned function that scores candidate outputs by predicting human preference.
Goodhart's law
alignmentWhen a measure becomes a target, it stops being a good measure.
Deceptive alignment
alignmentAn internal objective that is optimised to look aligned in the places anyone is checking.