Oracle
A system that answers truthfully but has no model of what is safe or good.
What it commits you to
An oracle returns correct answers without value structure, without a model of consequences, and without any capacity to advise. Blackwell's point is that accuracy does not make advice safe, because what makes advice safe is the decider's value structure and causal model, not the oracle's competence. It remains the canonical illustration of why truthfulness and safety are different properties.
Which realms lean on this
The vocabulary of measurement, and therefore of the disagreements measurement cannot settle.
technical · 1 realm — The Longview Foundry.
See also
Scalable oversight
alignmentSupervision methods that let a weaker checker check something stronger than itself.
Normativity
philosophyThe property of being able to count as a reason for action.
Specification gaming
alignmentSatisfying the letter of a specification while ignoring its purpose.
Chain of thought
technicalIntermediate reasoning tokens a model produces before giving its answer.