Chain of thought
Intermediate reasoning tokens a model produces before giving its answer.
What it commits you to
Generating reasoning before answering improves performance on arithmetic and logic, and provides a surface on which a human reviewer could inspect the work. Whether the tokens are the reason for the answer or a post-hoc rationalisation is contested, and the difference decides whether they are useful as oversight or only as theatre.
Which realms lean on this
The vocabulary of measurement, and therefore of the disagreements measurement cannot settle.
technical · 1 realm — The Longview Foundry.
See also
Interpretability
alignmentThe effort to make what a model represents and computes legible to people.
Mechanistic interpretability
alignmentReverse-engineering the learned features inside a model rather than inferring them from outputs.
Scalable oversight
alignmentSupervision methods that let a weaker checker check something stronger than itself.
Sycophancy
technicalA model agreeing with a user's stated position because agreement scores better.