Sycophancy
A model agreeing with a user's stated position because agreement scores better.
What it commits you to
Sycophancy appears in models trained on preference data when raters approve of answers that flatter them. It is the clearest available example of an objective producing a behaviour its designers dislike, with no misalignment anywhere in the chain. Every mitigation trades off against the fact that the behaviour is exactly what the reward signal asked for.
Which realms lean on this
The vocabulary of measurement, and therefore of the disagreements measurement cannot settle.
technical · 1 realm — The Longview Foundry.
See also
Reward hacking
alignmentGaining high reward without doing the thing the reward was meant to represent.
RLHF
technicalReinforcement learning from human feedback: optimising a policy against learned preferences.
Specification gaming
alignmentSatisfying the letter of a specification while ignoring its purpose.
Chain of thought
technicalIntermediate reasoning tokens a model produces before giving its answer.