TRPO

TRPO, short for Trust Region Policy Optimization, is a reinforcement learning algorithm designed to stabilize policy updates. It optimizes a stochastic policy by enforcing that the new policy does not deviate too much from the previous one, measured by the Kullback–Leibler (KL) divergence. The core idea is to maximize the expected return while staying within a defined trust region around the old policy.

The method formulates a constrained optimization problem: maximize a surrogate objective that estimates improvement in expected return, subject to a KL divergence constraint between the updated policy and the old policy. In practice, the problem is approximated by linearizing the objective and quadraticizing the constraint, which leads to a trust-region update direction in the policy parameter space. This direction is computed using a natural-gradient approach involving the Fisher information matrix. Conjugate gradient methods are used to solve for the update direction efficiently, leveraging a Fisher-vector product to avoid explicit construction of large matrices.

A line search ensures that the updated policy achieves an actual improvement in the surrogate objective while respecting the KL bound. The result is a policy update that, under the method’s approximations, guarantees monotonic improvement of performance.

TRPO is an on-policy, model-free algorithm commonly applied with neural network policies, such as Gaussian policies for continuous actions. It tends to be more stable than standard policy-gradient methods but is relatively computationally intensive and more complex to implement. It influenced subsequent policy optimization methods, including Proximal Policy Optimization (PPO), which aims to retain TRPO’s stability with simpler mechanics. Introduced by Schulman and colleagues in 2015, TRPO remains a foundational approach in policy optimization research.