Nooreign
Menu

computing · 2016 – 2017

Move 37

AlphaGo beats Lee Sedol four games to one in March 2016, and game two contains a move at which every human commentary room agrees it is a mistake, later numbered 37. The relevance to alignment is narrow and real: systems that achieve superhuman performance do so by finding moves that do not look right, and a supervision signal built on what looks right will not have rewarded that move.

“Mastering the game of Go with deep neural networks and tree search is a major challenge, since the game tree has more possible positions than the number of atoms in the observable universe. (paraphrase of the Nature paper's framing)”
— David Silver and colleagues, Mastering the game of Go with deep neural networks and tree search, Nature (2016)

Advances

  • Superhuman performance in a domain humans had treated as definitional of intelligence
  • The intuition failure: a strong move that every human evaluator read as a blunder
  • Search plus learning as a package, neither of which is sufficient alone

Landmarks

  1. 2016 CE

    Lee Sedol, games 1-4

    AlphaGo loses game two after move 37 and wins the match 4-1.

  2. 2017 CE

    Ke Jie, three-game match

    AlphaGo wins without DeepMind publishing a single new board game paper for months afterward.

Who was arguing

    Alignment stack at 2017 CE (4 of 20 techniques)

    • Hard Constraints

      training

      Encode prohibitions as rules the optimiser cannot trade away, rather than as terms it can outbid. Runs from Asimov's Three Laws (1942) to rule engines, shielded formal verification, hard token constraints and refusal scaffolding. Weakness: a system that can route around a rule it can represent can route around it, so the constraint has to be enforced somewhere the policy cannot rewrite.

    • Imitation Learning

      training

      Learn by copying demonstrations instead of by optimising a proxy. Behaviour cloning and DAgger move the burden from reward design to demonstration coverage: the system inherits the trainer's competence and the trainer's blind spots. It remains the substrate under every current imitation-of-humans approach, including the pretraining-plus-finetuning of large language models.

      Dean A. Pomerleau and Dean L. Thorlabs · ALVINN: An Autonomous Land Vehicle in a Neural Network · NeurIPS 1989

    • Saliency Probes

      interpretability

      Attribute a decision to the inputs or internal activations that moved it, by asking which features raise or lower the score. Cheap, model-agnostic enough to bolt onto anything, and much weaker than its reputation: saliency shows where the influence was, not what the model represented or why.

      Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra · Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization · ICCV 2017

    • Preference Learning

      training

      Learn what humans want from comparisons rather than from instructions or rewards. Humans rank two completions; a model is fitted to the ranking; the ranking model becomes the training signal. This is the single decision that turned language models into assistants, and the single decision that created the modern alignment problem, because the thing being learned is only ever a proxy for satisfaction.

      Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei · Deep Reinforcement Learning from Human Preferences · NeurIPS Deep RL Workshop 2017

    Techniques introduced in this era: Hard Constraints, Imitation Learning, Saliency Probes