ML//RL//AlphaGo

AlphaGo is a Go-playing system built by DeepMind that combines Monte Carlo tree search with neural networks for move selection and position evaluation, and in 2016 it beat Lee Sedol. AlphaGo Zero later removed human game data and learned through self-play.

AlphaGo is a Go-playing system built by DeepMind that combines Monte Carlo tree search with neural networks for move selection and position evaluation, and in 2016 it beat Lee Sedol. AlphaGo Zero later removed human game data and learned through self-play.

Search and learning divide the work: the network compresses experience into useful priors; tree search (Monte Carlo tree search) spends computation on the position currently in front of it. The networks were convolutional, and calling AlphaGo a ConvNet would still hide most of what made it play: the search at move time and the reinforcement learning that trained it. The same caution applies to calling a reasoning system just an LLM (test-time compute).

Self-play manufactures a curriculum whose difficulty rises with the agent. The opponent is always approximately as capable as the learner because it is another version of the learner.

The deeper result went beyond Go: it made policy improvement, value estimation, self-play and planning, the separate pieces of reinforcement learning, work as one system, and showed that learned intuition could make combinatorial search practical in spaces where brute force was absurd.