genie · Anonymous Middle Eastern folktale, as used in AI-control writing
Genie Sweep
When does a requester lose the ability to foresee the consequences well enough to make the request responsibly?
A single request is made to an agent that will fulfil it exactly, has no interests, cannot be consulted afterwards, and gives no warning. The sweep varies the number of wishes, the irreversibility of the effects, the requester's ability to model second-order consequences, and how exactly the fulfilment language is written.
first published 1949 · The genie of folklore, pressed into service as a modern control problem: an agent that grants exactly what was asked, without consultation or warning
The bench
computing…What the sweep shows
- Higher foresight improves the expected outcome almost linearly and barely moves the tail risk, which is the formal version of the reason no amount of careful planning makes unbounded power safe.
- Exact-wording fulfilment produces unintended outcomes at a rate proportional to specification slack and entirely independent of how well intention was formed — the specifier's care is not visible to the fulfiller.
- Additional wishes make outcomes worse rather than better, because each one is made under the distorted information the previous wish produced.
- The results support the Stoic reading rather than the utilitarian one: since consequences dominate and cannot be foreseen, the only variable the requester actually controls is the scope of what they ask for.
Sources
- — Bostrom, Superintelligence: Paths, Dangers, Strategies (2014), on goal misgeneralisation
- — Amodei et al., Concrete Problems in AI Safety (2016)
- — Epictetus, Enchiridion (c. 125)