Existential risk
A risk that permanently curtails humanity's future, including by ending it.
What it commits you to
Existential risks are those that either extinguish humanity or permanently and severely constrain its long-term prospects. They are defined by scale and irreversibility rather than by probability, which is why standard expected-value frameworks handle them badly. Whether AI counts as one is a live dispute: the argument is not that catastrophe is likely but that it is unbounded, and unbounded is not a probability.
Which realms lean on this
Every realm has to keep a system honest. These are the terms that say out loud what the other kinds keep implicit.
alignment · 5 realms — The Benthamite Foundry, The Existential Wilds, The Longview Foundry, The Stoic Ramparts, The Ubuntu Weave.
Canon that discusses this
modern · 1951
The Rebel
Camus argues that rebellion commits itself to the future rather than denying the present, and that revolutions that stop at negation produce a new orthodoxy. The final section's account of rebellion without a cause is the bleakest thing he wrote, and it is the part anyone deploying a revolt should read first. The excerpt below paraphrases the argument rather than quoting the 1951 book.
modern · 1946
Existentialism Is a Humanism
Sartre wrote it as a lecture in Paris in October 1945 and it was published the following year; the title is the argument, and the defence of freedom as responsibility for one's own situation is the part that survives. He also argues here against the charge that existentialism places man above nature. The excerpt below paraphrases the lecture rather than quoting the 1946 text.
See also
Precautionary principle
ethicsWhen serious harm is possible and the evidence is uncertain, act to avoid it.
Normative uncertainty
ethicsUncertainty not about what is true but about what ought to be valued.
FOOM
alignmentFoom: a hypothetical scenario of fast, irreversible capability blowup.
Imperative of responsibility
philosophyThe duty to weigh the full, indefinite consequences of an action before taking it.