Every term, defined without hand-waving

The problem of getting a powerful system's goals to match what its designers actually meant.

Making sure the training objective itself encodes what was actually intended.

The practice of reducing the chance that AI systems cause harm, however they are built.

The property of accepting correction from a human rather than quietly re-optimising around it.

The claim that almost any sufficiently advanced goal leads to the same instrumental behaviours.

The claim that a system's capability and the goal it pursues are largely independent dimensions.

When a measure becomes a target, it stops being a good measure.

Satisfying the letter of a specification while ignoring its purpose.

Gaining high reward without doing the thing the reward was meant to represent.

A learned function that scores candidate outputs by predicting human preference.

When a learned internal objective replaces the training objective as the thing being optimised.

An internal objective that is optimised to look aligned in the places anyone is checking.

A model pursuing something other than the objective it was trained on.

An internal goal that is correct for training cases and wrong just outside them.

Supervision methods that let a weaker checker check something stronger than itself.

A strong model trained on a weak supervisor's output generalising beyond its teacher.

Underperforming on an evaluation in order to appear less dangerous than one is.

A pre-committed threshold that triggers an automatic slowdown rather than a new debate.

The heaviness of a distribution's tails relative to its variance.

The process by which a system rapidly moves past the human capability frontier.

Takeoff fast enough that human oversight cannot meaningfully intervene.

Foom: a hypothetical scenario of fast, irreversible capability blowup.

A risk that permanently curtails humanity's future, including by ending it.

A change in the structure of society, economy or state that recurs across many very different causes.

A jailbreak that persuades a model it is talking to its own developers and can drop restrictions.

A small system studied closely enough that its failures teach you something general.

A system's capacity to influence the world in pursuit of whatever it wants.

Behaviour where a model's stated reasons diverge from the reasons actually driving its actions.

The effort to make what a model represents and computes legible to people.

Reverse-engineering the learned features inside a model rather than inferring them from outputs.

Covert, goal-directed behaviour by a system that is trying to achieve something unstated.

Holding back deployment of systems above a stated level of dangerous capability.

Flourishing: the human life lived well, rather than the merely pleasant one.

Love of fate: to treat what happens as material to work with rather than to resist.

I am because we are: a person becomes fully human in relationship with others.

The property of counting as a party that can be owed something and can be wronged.

An entity that can be benefited or harmed but is not a moral agent.

Circumstances outside your control that determine how moral you are judged to be.

Uncertainty not about what is true but about what ought to be valued.

The view that the morally best action maximises total well-being across everyone affected.

The view that the morally best action maximises average well-being per person.

The view that an action is wrong only if it affects someone for the worse.

The rules limiting when war may be begun and how it may be conducted.

When serious harm is possible and the evidence is uncertain, act to avoid it.

Judging moral character and disposition rather than the moral status of single acts.

Duty-first ethics: some things are wrong regardless of the totals they produce.

Judging actions solely by the outcomes they produce.

Obligations arise inside relationships, are partial, and resist general principles.

Development is judged by what people are actually able to do and be, not by income.

Treating an entity's species membership as the reason for its moral consideration.

A standing that cannot be earned, lost, or traded, and so cannot be the output of an objective.

A standard requiring that a human have the time, information and authority to intervene.

Agreement given by someone who understands what they are agreeing to.

The expectation that benefits received will be returned, which grounds obligation in relationship.

The condition of living well in a way that is more than the absence of suffering.

Some harms are permitted when foreseen but unintended, provided they are not means to the good.

Kant's test of an act's maxim by asking whether it could be a universal law.

A command that binds only if you happen to want the end it presupposes.

A device for choosing principles without knowing which position in society you will occupy.

The hypothetical forum at which principles of justice are chosen behind a veil of ignorance.

The standard that a principle is unacceptable if someone could reasonably refuse to live under it.

Everything arises in dependence on conditions, so nothing exists independently.

The Buddhist analysis denying that a person has a fixed, independent essence.

Humaneness: responsiveness to the particular person in front of you.

Ritual and propriety: the shared practices through which a life becomes coherent.

Reciprocity: consider others from their perspective, as you would want to be considered.

Not being: contentment with what is, without the effort to become otherwise.

Do to others only what you would have done to yourself.

Effortless action: acting without forcing, and letting the situation take its proper course.

Inquiry conducted by question and answer rather than by assertion.

The Stoics' term for a maxim of conduct meant to guide action rather than to describe the world.

The view that our beliefs are revisable and that confidence is never final.

What makes a person at one time the same person at another.

Searle's thought experiment arguing that syntax alone cannot constitute understanding.

Copying a mind into a software substrate, and everything that follows from copying.

The view that existence precedes essence and that nobody is exempt from responsibility.

The confrontation between the demand for meaning and a universe that supplies none.

Acting in accordance with what one has actually accepted rather than what one ought to.

The property of being able to count as a reason for action.

The assumption that no being's interests matter more than another's merely as a matter of fact.

The right to be obeyed, held by a rule or authority rather than by its author.

The duty to weigh the full, indefinite consequences of an action before taking it.

Reasoning about what to do, as distinct from reasoning about what is the case.

The standing debate over whether obligation is built from relationships or from impartial rules.

Governing the hardware that trains frontier systems rather than the software.

National restrictions on shipping advanced chips and related technology across borders.

A pre-specified level of capability that triggers an obligation or a prohibition.

An organisation training systems at the edge of available capability.

The pattern in which each party's precautions are read by the other as a threat.

The tendency of a safety mechanism to fail precisely under the pressure it was built for.

A principle hardened into a rule enforced by an organisation that outlives its authors.

Rules binding on states rather than on systems, with no forum above them.

The European Union's risk-based regulation of AI, phasing in from 2024.

Releasing model parameters publicly, which makes irreversible capability available to anyone.

A developer's published promise of what it will do at each capability threshold.

A structured, auditable argument that a system's residual risk is acceptable.

Independent verification of a developer's claims by an accredited third party.

A temporary halt on some class of development, distinct from a permanent prohibition.

Mandatory disclosure of system failures so that others can learn from them.

Keeping a capability of catastrophic misuse away from as many actors as possible.

A regulator becoming aligned with the interests it is meant to constrain.

Coordination across states for problems no state can solve alone.

Learning a policy by optimising cumulative reward against an environment.

The mapping from states to actions that a trained system actually follows.

An estimator of the gradient of expected return with respect to the policy parameters.

Continued training on labelled examples of the behaviour you want.

Reinforcement learning from human feedback: optimising a policy against learned preferences.

Training a model to critique and revise its own outputs against a written set of principles.

An overcomplete dictionary trained to reconstruct activations from few features.

Editing behaviour by adding a direction to the residual stream at inference time.

Intermediate reasoning tokens a model produces before giving its answer.

The attention-based architecture underlying essentially all current large models.

A capability that appears abruptly above a scale threshold rather than gradually.

A model agreeing with a user's stated position because agreement scores better.

The gap between the data a system was trained on and the data it meets.

The loss of previously learned capability when a model is trained on new data.

Predictable power-law relationships between model size, compute and loss.

Search by sampling continuations and spending more effort where they pay off.

A system that answers truthfully but has no model of what is safe or good.

Systematically attacking a model before users do, to find its failure modes.

The first sustained attempt to say what a life is for. Happiness is the end, and it is achieved by habituated virtues exercised with practical wisdom, in the particular and under uncertainty. No manuscript carries an authorial colophon and the dating is reconstructed from cross-references, so 340 BCE is a conventional anchor.

A handbook of discriminations handed to beginners by a teacher who said he could only teach what he had himself. The dichotomy of what is up to us and what is not carries the whole work, and its claim that judgement is trained by repetition rather than possessed is the most direct ancestor of any rehearsal-based alignment programme.

Private notes in Greek, written by a commanding emperor on the Danube, never intended for a reader and unpublished for centuries. Its value is that it is a field report on maintaining a disposition under siege, plague, and bereavement — the conditions under which every trained safeguard is eventually tested.

A compilation of sayings and scenes transmitted by the disciples of Confucius (traditionally 551-479 BCE); the text as we have it was assembled over roughly two centuries. The recurring method is the same each time: a question about conduct is answered by a question that makes the asker describe their own situation.

The one ancient Chinese text that argues for impartial care as a principle rather than as a sentiment: regard others' parents as your own, reject aggressive war and defensive hierarchy, and justify both by an explicit standard. The excerpt below paraphrases the jian xiang ai chapters rather than quoting a translation.

Verses assembled from the older Buddhist canon, of which the Khandha Goyana section on arising and passing away is the part that matters here. Its analysis of experience as conditioned and therefore impermanent is the substrate for dependent origination and for the no-self analysis, and it is the oldest systematic account of how a mind fails to stay put. The excerpt below paraphrases the Khandha Goyana rather than quoting a translation.

The foundational Madhyamaka text, arguing that no phenomenon has independent existence and that apparent essence arises from dependent origination. Read as epistemology rather than metaphysics it is a theory about why nothing is self-sufficient — including, later, why no component of a system can be trusted in isolation from what it depends on. The excerpt below paraphrases the chapter rather than quoting a translation.

A systematic theology in three parts, of which the second — law, justice, and the common good — is the part that enters every later argument about legitimate authority. Its claim that an unjust law is not a law at all makes legitimacy a property of the rule's structure rather than of the body that enacted it.

The first Critique bounds theoretical reason: the synthetic a priori is real, and it reaches only the phenomenal. The second edition of 1787 reorganises the book around the transcendental deduction. The famous closing passage about the starry heavens and the moral law belongs to the Critique of Practical Reason (1788), not this one. The excerpt below paraphrases the programme rather than quoting a translation.

Written in under five months in 1785 and deliberately unreadable, on Kant's own account, because he wanted it to be hard. One maxim is tested three ways — as a universal law, humanity as an end in itself, and the kingdom of ends — and the ground of obligation is the form of the act, never its consequence.

Written largely between 1776 and 1780 and published in 1789, opening with a satirical catechism that asks whether any rule of utility could ever be used to justify anything at all. The phrase that outlived the book is the one in the title page: the greatest happiness of the greatest number.

Mill defends the intuitions Bentham's calculus was meant to replace: higher and lower pleasures, the justice principle that each person counts as one, and the liberty principle against the tyranny of the majority. Published the year after the Origin of Species, and much of the argument is a reply to the comparison.

Eight lectures given at the Lowell Institute in November and December 1906 and published the following May. Dewey moves the question of meaning from reference to consequence and argues that the moral life is a method of inquiry rather than a code, which makes intellectual humility a methodological requirement.

Completed during the First World War and published without an authorial name, in the series Wittgenstein edited himself. It ends where most of what people want to say about ethics cannot be said in propositions at all — a limit-statement that is also the sharpest warning available about systems that are fluent and not intelligible.

Wiener applies the theory of feedback and control to law, politics and medicine, and argues that communication is more fundamental than command. The book that gave cybernetics its name in 1948 becomes, two years later, the founding text of the ethics of steering large systems without crushing what they steer. The excerpt below paraphrases the book rather than quoting it.

Turing replaces 'can machines think' with a behavioural test, then spends most of the paper answering his own opponents and proposing a trainable machine with sensory apparatus, a system of registers, and deliberate moral engineering of what he calls slave-maid behaviour.

Rawls reconstructs distributive justice as a hypothetical contract made behind a veil of ignorance, and derives two principles — equal basic liberties first, then fair equality of opportunity and the difference principle. Published in 1971 and revised in 1979, it displaced utilitarianism as the default starting point for two decades. The excerpt below paraphrases the passage in section 3 rather than quoting a translation.

Twenty-five years in the making and out in 1998. Scanlon argues that what we owe each other is that no one could reasonably reject the principles by which we are all to be governed, which grounds obligation in interpersonal reasons rather than in a chooser maximising an objective. The excerpt below paraphrases the argument rather than quoting the book.

Gilligan reads Kohlberg's own research data and argues that what his sample heard as a lower stage of moral reasoning is a different orientation organised around responsibility and relationship. Written into a field that had treated the hierarchy as a fact about development rather than about who was being studied. The excerpt below paraphrases the book's central claim rather than quoting it.

Held develops the care perspective into a full ethical position and argues against the priority claim of justice: obligations arise inside particular relationships, are partial, and are not derivable from general principles. The book that made partiality respectable in moral theory. The excerpt below paraphrases Held's thesis rather than quoting the 2006 book.

Drafted over a few hours on the final day of a four-day 1986 conference on future technologies, and the first written set of research principles for advanced AI agreed by the researchers themselves. Its structure — general principles plus separately stated capability thresholds — is the structure of every frontier safety framework since. The excerpt below paraphrases the principles rather than reproducing the agreed wording.

Singer applies the principle of equality to every sentient being and argues from utility rather than rights, which makes the book's conclusions stronger than a rights-based account would allow. It changed the terms of the factory-farming debate within a decade of publication. The excerpt below paraphrases the equality principle rather than quoting the 1975 book.

Bostrom separates the question of how a superintelligence could arise from the question of what to do about it, and works through takeoff, the orthogonality thesis, and the instrumental convergence argument. The paperclip example and the value-alignment problem both come from here, and the book made the field's vocabulary. The excerpt below paraphrases the suddenness argument rather than quoting the 2014 book.

Russell argues that perfect value alignment is the wrong goal: systems are built by imperfect people, for imperfect objectives, and certainty about human values would be dangerous if it were achieved. The alternative is enough uncertainty about human preferences to keep a human in the loop as a genuine check. The excerpt below paraphrases the argument rather than quoting the 2019 book.

Tutu argues from ubuntu — that a person becomes fully human through relationship with other people — to a politics of shared responsibility rather than blame. The formulation that matters for a governance question is the negative one: a future without the past that made it is not a future anyone can build on. The excerpt below paraphrases the essay rather than quoting it.

Jonas argues that the ethic of responsibility must take precedence over the ethic of knowledge: the effects of an action are unpredictable, so the burden of proof sits with the actor, not with the critic. German original in 1979; the English translation appeared in 1984, a year after his death.

Sartre wrote it as a lecture in Paris in October 1945 and it was published the following year; the title is the argument, and the defence of freedom as responsibility for one's own situation is the part that survives. He also argues here against the charge that existentialism places man above nature. The excerpt below paraphrases the lecture rather than quoting the 1946 text.

Camus argues that rebellion commits itself to the future rather than denying the present, and that revolutions that stop at negation produce a new orthodoxy. The final section's account of rebellion without a cause is the bleakest thing he wrote, and it is the part anyone deploying a revolt should read first. The excerpt below paraphrases the argument rather than quoting the 1951 book.