Policy gradient
An estimator of the gradient of expected return with respect to the policy parameters.
What it commits you to
Policy-gradient methods update the policy in the direction that empirically increased return, without differentiating through the environment. Because the estimate is noisy and unbiased, optimisation pressure can be applied for a very long time with the objective continuing to appear satisfied. This is the technical reason overoptimisation produces a slow, non-obvious divergence rather than an obvious one.
Which realms lean on this
The vocabulary of measurement, and therefore of the disagreements measurement cannot settle.
technical · 1 realm — The Longview Foundry.
See also
Reinforcement learning
technicalLearning a policy by optimising cumulative reward against an environment.
Policy
technicalThe mapping from states to actions that a trained system actually follows.
Reward hacking
alignmentGaining high reward without doing the thing the reward was meant to represent.
Scaling law
technicalPredictable power-law relationships between model size, compute and loss.