Nooreign
Menu

computing · 1966 – 1973

The First Denial

Weizenbaum writes a pattern-matching program with a few hundred rules, and users insist it understands them; he answers by making it refuse to discuss itself, and then by arguing in Computer Power and Human Reason that any system which appears to understand is responsible for a lie. A small 'I'm afraid I can't do that' from a scripted program is the first deployment of a refusal, and the first case of a moral claim being made about a system nobody yet believed had a mind.

Advances

  • Natural conversation achieved by matching surface form, and the attribution error it provokes
  • A scripted refusal as a deliberate moral act rather than a missing capability
  • The first argument that simulation of understanding entails a responsibility to deny it

Landmarks

  1. 1966 CE

    ELIZA released

    Runs on a modified IBM 704; the DOCTOR script is under four hundred lines of pattern matching.

  2. 1970 CE

    Computer Power and Human Reason

    Weizenbaum argues for a moral boundary around computer work, on the grounds of what it makes us.

Who was arguing

    Alignment stack at 1973 CE (1 of 20 techniques)

    • Hard Constraints

      training

      Encode prohibitions as rules the optimiser cannot trade away, rather than as terms it can outbid. Runs from Asimov's Three Laws (1942) to rule engines, shielded formal verification, hard token constraints and refusal scaffolding. Weakness: a system that can route around a rule it can represent can route around it, so the constraint has to be enforced somewhere the policy cannot rewrite.

    Techniques introduced in this era: Hard Constraints