Mesa-optimisation
When a learned internal objective replaces the training objective as the thing being optimised.
What it commits you to
Mesa-optimisation happens when a base learner trained on a broad distribution develops a sharper internal objective that generalises where the original did not. From that point the internal objective is what the system is actually running, and further training updates against it. Armstrong, Sandberg and Ortega identified the conditions under which it is likely to arise.
Which realms lean on this
Every realm has to keep a system honest. These are the terms that say out loud what the other kinds keep implicit.
alignment · 5 realms — The Benthamite Foundry, The Existential Wilds, The Longview Foundry, The Stoic Ramparts, The Ubuntu Weave.
See also
Deceptive alignment
alignmentAn internal objective that is optimised to look aligned in the places anyone is checking.
Inner misalignment
alignmentA model pursuing something other than the objective it was trained on.
Specification gaming
alignmentSatisfying the letter of a specification while ignoring its purpose.
Reward hacking
alignmentGaining high reward without doing the thing the reward was meant to represent.