policygradient

Policy gradient refers to a class of methods in reinforcement learning that optimize a parameterized stochastic policy directly by gradient ascent on the expected return. The policy is denoted pi_theta(a|s), giving the probability of taking action a in state s under parameters theta. The objective is J(theta) = E_pi_theta [ sum_{t=0}^∞ γ^t r_t ], or the equivalent episodic return. The policy gradient theorem provides the gradient of J with respect to theta as ∇θ J(θ) = E_pi_theta [ ∇θ log pi_theta(a|s) Q^π(s,a) ], where Q^π is the action-value function under the current policy.

In practice, the gradient is estimated from interactions with the environment. The REINFORCE algorithm uses Monte Carlo samples and yields ∇θ J ≈ ∑ ∇θ log pi_theta(a_t|s_t) G_t, where G_t is the return from time t. This estimator has high variance, which can be reduced by subtracting a baseline b(s) or by using an advantage function A^π(s,a) = Q^π(s,a) − V^π(s). The resulting estimator ∇θ J ≈ E[ ∇θ log pi_theta(a|s) A^π(s,a) ] forms the basis of actor-critic methods.

Actor-critic methods train a critic to estimate V^π or Q^π and update the policy (actor) with the advantage signal. For continuous action spaces, stochastic policies with Gaussian outputs or other parametric forms are common; deterministic policy gradient methods use the gradient ∇θ J ≈ E[ ∇θ μ_θ(s) ∇_a Q^π(s,a) |_{a=μ_θ(s)} ]. Deep RL extends policy gradients with neural networks, enabling complex tasks; notable variants include PPO, TRPO, A2C/A3C, and DDPG/DDPG-like off-policy approaches.

Considerations include variance reduction, exploration, sample efficiency, and stability. Regularization terms such as entropy bonuses encourage exploration. The term policy gradient often refers to the broader family of gradient-based optimization methods for stochastic policies, with many practical algorithms combining these ideas.