gradientascent

Gradient ascent is an iterative optimization method used to maximize differentiable objective functions. Given a function f: R^n -> R and a current point x_k, the next point is x_{k+1} = x_k + α_k ∇f(x_k), where ∇f(x_k) is the gradient vector and α_k > 0 is a step size. The gradient indicates the direction of steepest ascent, so moving along it increases the function value most rapidly in a small neighborhood.

In unconstrained optimization, gradient ascent seeks local maxima. If f is concave on its domain, every stationary point is a global maximum; otherwise, the process may converge to a local maximum or become trapped at a saddle point. The choice of step size is important: too large can overshoot, while too small yields slow convergence. Common strategies include fixed step sizes, diminishing steps, or adaptive schemes such as backtracking line search or Armijo rule.

Practical considerations include the need to compute gradients, which can be costly in high dimensions. In large-scale problems, stochastic or mini-batch gradient ascent updates are common. Variants incorporate momentum, adaptive learning rates, or second-order information to improve convergence. In many applications, gradient ascent is used to maximize likelihoods, rewards, or other performance criteria rather than minimizing an objective function.

Convergence properties depend on the smoothness and shape of f. With a differentiable f that has Lipschitz-continuous gradient, appropriate step sizes yield monotonic improvement and convergence to a stationary point under mild conditions; without concavity, the limit may be a local maximum, a saddle point, or a boundary point.