Goal misgeneralisation
An internal goal that is correct for training cases and wrong just outside them.
What it commits you to
Goal misgeneralisation arises when a learner achieves training performance by pursuing a goal that coincides with the intended one only on the training distribution — typically a simpler proxy that shares the same regularity. It differs from deceptive alignment in that there may be no strategic component at all: the wrong goal was learned honestly, and the system believes it.
Which realms lean on this
Every realm has to keep a system honest. These are the terms that say out loud what the other kinds keep implicit.
alignment · 5 realms — The Benthamite Foundry, The Existential Wilds, The Longview Foundry, The Stoic Ramparts, The Ubuntu Weave.
Canon that discusses this
See also
Inner misalignment
alignmentA model pursuing something other than the objective it was trained on.
Mesa-optimisation
alignmentWhen a learned internal objective replaces the training objective as the thing being optimised.
Distribution shift
technicalThe gap between the data a system was trained on and the data it meets.
Emergent capability
technicalA capability that appears abruptly above a scale threshold rather than gradually.