offpolicy

Off-policy learning refers to reinforcement learning methods where the agent learns about a target policy while following a behavior policy that may be different. This separation allows using data collected under other policies or from past experiences, enabling offline data use and improved sample efficiency.

Key idea: The target policy is optimized, while the behavior policy determines actions. The mismatch between data distribution and the target policy is mitigated with importance sampling or other correction techniques. Off-policy methods can reuse transitions stored in replay buffers, decoupling data collection from learning.

Examples: Q-learning is a canonical off-policy algorithm. Deep Q-Networks extend Q-learning to function approximation with experience replay and a separate target network. Off-policy actor-critic methods—such as Deep Deterministic Policy Gradient (DDPG), Twin Delayed DDPG (TD3), and Soft Actor-Critic (SAC)—learn a policy from stored experiences.

Advantages and challenges: Off-policy learning can leverage large and diverse datasets and support offline RL, but introduces higher variance and potential instability due to distribution mismatch, bootstrapping, and function approximation. Techniques to improve stability include replay buffers, target networks, clipped importance sampling weights, and regularization terms.

Off-policy evaluation and learning: In addition to optimizing a policy, off-policy evaluation estimates a policy’s performance using data from other policies, via importance sampling, weighted importance sampling, fitted-Q evaluation, or model-based approaches.

Applications: Widely used in robotics, video games, and simulated environments where offline data or replay is available.

See also: on-policy learning; policy gradient; Q-learning; deep reinforcement learning; off-policy evaluation.