Distribution shift
The gap between the data a system was trained on and the data it meets.
What it commits you to
Any distribution shift degrades performance, and the degradations are worst exactly where the new inputs are least like the training ones. In alignment this is where goal misgeneralisation and deceptive alignment live, since a model that behaves well in-distribution has no reason to behave well anywhere else. It is also why deployment on a wider distribution than training is the standard stress test.
Which realms lean on this
The vocabulary of measurement, and therefore of the disagreements measurement cannot settle.
technical · 1 realm — The Longview Foundry.
See also
Goal misgeneralisation
alignmentAn internal goal that is correct for training cases and wrong just outside them.
Deceptive alignment
alignmentAn internal objective that is optimised to look aligned in the places anyone is checking.
Emergent capability
technicalA capability that appears abruptly above a scale threshold rather than gradually.
Policy
technicalThe mapping from states to actions that a trained system actually follows.