Policy
The mapping from states to actions that a trained system actually follows.
What it commits you to
In reinforcement learning the policy is the artefact, and the objective is merely what shaped it. Separating the two is the central move of alignment: once behaviour is understood as the output of optimisation rather than as the expression of an intention, every question becomes about the objective and the training distribution. The language model field's habit of discussing models as though they had intentions is the thing this term is meant to disrupt.
Which realms lean on this
The vocabulary of measurement, and therefore of the disagreements measurement cannot settle.
technical · 1 realm — The Longview Foundry.
See also
Reinforcement learning
technicalLearning a policy by optimising cumulative reward against an environment.
Policy gradient
technicalAn estimator of the gradient of expected return with respect to the policy parameters.
Monte Carlo tree search
technicalSearch by sampling continuations and spending more effort where they pay off.
Specification gaming
alignmentSatisfying the letter of a specification while ignoring its purpose.