DQN

DQN, or Deep Q-Network, is a deep reinforcement learning algorithm developed by DeepMind and published in 2015. It combines Q-learning with deep neural networks to approximate the action-value function Q(s,a). DQN can learn control policies from high-dimensional sensory input such as raw video frames, enabling end-to-end learning without hand-crafted features.

Core mechanisms of DQN include using a convolutional neural network to map states to Q-values for each possible action. For stability, it maintains an experience replay buffer and updates the network using mini-batches drawn from this buffer. It also employs a separate target network to compute targets y = r + gamma max_a' Q_target(s', a'), with the target network updated periodically to decouple targets from current network parameters. Action selection is typically done via an epsilon-greedy policy. Training minimizes the mean-squared error between current Q-values and the computed targets.

In terms of performance, DQN demonstrated strong results on Atari 2600 games, achieving human-level performance on many titles using only raw pixel input and game score as feedback. It established a practical baseline for applying deep learning to reinforcement learning. Limitations include sample inefficiency, significant computational demands, instability in some domains, and challenges with continuous action spaces.

Several extensions have been proposed to address these limitations, including Double DQN (reducing overestimation bias), Dueling DQN (separating state-value and advantage estimates), and Prioritized Experience Replay (improving sample efficiency). Aggregated approaches like Rainbow combine multiple improvements to enhance stability and performance across tasks.