Reinforcement learning
Learning a policy by optimising cumulative reward against an environment.
What it commits you to
Reinforcement learning optimises a policy for expected return, which makes it the only mainstream method that produces systems whose behaviour is determined by an objective. That is its attraction for builders and its problem for critics: behaviour is determined by the objective whether or not anyone intended the resulting behaviour. Almost every alignment failure is an instance of this.
Which realms lean on this
The vocabulary of measurement, and therefore of the disagreements measurement cannot settle.
technical · 1 realm — The Longview Foundry.
See also
Policy
technicalThe mapping from states to actions that a trained system actually follows.
Policy gradient
technicalAn estimator of the gradient of expected return with respect to the policy parameters.
Reward hacking
alignmentGaining high reward without doing the thing the reward was meant to represent.
Specification gaming
alignmentSatisfying the letter of a specification while ignoring its purpose.