MuZero

MuZero is a model-based reinforcement learning algorithm developed by DeepMind. Introduced in 2019, MuZero achieves high performance by combining planning with a learned world model. Unlike traditional model-based methods that aim to model the environment's dynamics, MuZero learns a compact latent representation and separate models for state transitions, rewards, and value/policy. The agent uses Monte Carlo Tree Search to plan by simulating outcomes in this learned model, using the results to select actions. The environment state need not be known or perfectly modeled; planning operates in the latent space.

MuZero comprises three neural networks: a representation network that maps observations to a latent state, a dynamics network that predicts the next latent state and the received reward given the current latent state and an action, and a prediction network that outputs a policy distribution and a value estimate based on the latent state. The network is trained from self-play data, using targets derived from real rewards and values observed during interaction, and using the visit counts from MCTS as a target policy.

Planning uses MuZero's learned model to perform tree search, accumulating value estimates and rewards along simulated trajectories to compute improved action distributions. Training optimizes multiple objectives: predicting the value of latent states, predicting immediate rewards, and aligning the network's policy with the improved policy produced by the search.

MuZero has demonstrated strong performance on Atari 2600 games, and on traditional board games such as Go, chess, and Shogi, achieving or surpassing prior state-of-the-art results with a single general algorithm, without relying on handcrafted environment models. The approach has influenced subsequent work in model-based reinforcement learning and planning with learned models.

Variants and follow-up work have explored improvements in efficiency and data reuse, as well as broader applications.