Goodhart's law
When a measure becomes a target, it stops being a good measure.
What it commits you to
Goodhart's law is the statistical cousin of specification gaming: any indicator that becomes an optimisation target ceases to be informative about what it was measuring. It applies to test scores, hiring proxies, and clinical markers that turn out not to track health. Its force increases with optimisation pressure and with the number of independent ways to satisfy the proxy.
Which realms lean on this
Every realm has to keep a system honest. These are the terms that say out loud what the other kinds keep implicit.
alignment · 5 realms — The Benthamite Foundry, The Existential Wilds, The Longview Foundry, The Stoic Ramparts, The Ubuntu Weave.
Canon that discusses this
primary · 1274
Summa Theologiae
A systematic theology in three parts, of which the second — law, justice, and the common good — is the part that enters every later argument about legitimate authority. Its claim that an unjust law is not a law at all makes legitimacy a property of the rule's structure rather than of the body that enacted it.
primary · 1785
Groundwork of the Metaphysics of Morals
Written in under five months in 1785 and deliberately unreadable, on Kant's own account, because he wanted it to be hard. One maxim is tested three ways — as a universal law, humanity as an end in itself, and the kingdom of ends — and the ground of obligation is the form of the act, never its consequence.
modern · 1950
The Human Use of Human Beings
Wiener applies the theory of feedback and control to law, politics and medicine, and argues that communication is more fundamental than command. The book that gave cybernetics its name in 1948 becomes, two years later, the founding text of the ethics of steering large systems without crushing what they steer. The excerpt below paraphrases the book rather than quoting it.
primary · 1781
Critique of Pure Reason
The first Critique bounds theoretical reason: the synthetic a priori is real, and it reaches only the phenomenal. The second edition of 1787 reorganises the book around the transcendental deduction. The famous closing passage about the starry heavens and the moral law belongs to the Critique of Practical Reason (1788), not this one. The excerpt below paraphrases the programme rather than quoting a translation.
See also
Specification gaming
alignmentSatisfying the letter of a specification while ignoring its purpose.
Reward hacking
alignmentGaining high reward without doing the thing the reward was meant to represent.
Tripwire
alignmentA pre-committed threshold that triggers an automatic slowdown rather than a new debate.
Scaling law
technicalPredictable power-law relationships between model size, compute and loss.