quasiNewtonin

Quasi-Newtonin, or quasi-Newton methods, are a family of iterative methods for unconstrained optimization that aim to minimize a differentiable function by updating an approximation to the Hessian matrix of second-order partial derivatives, or to its inverse. The core idea is to use gradient information from the current iterate to progressively improve a curvature estimate, avoiding the need to compute the exact Hessian at every step.

In a typical iteration, the current point xk has gradient gk = ∇f(xk). A search direction pk is chosen as the negative product of the inverse Hessian approximation Hk with the gradient, pk = -Hk gk (or equivalently using Bk as an approximation to the Hessian). A line search is usually performed to ensure sufficient decrease of f. After moving to xk+1, the gradient gk+1 is computed and the difference vectors sk = xk+1 − xk and yk = gk+1 − gk are formed. The Hessian or its inverse is updated to satisfy the secant equation Bk+1 sk = yk (or Hk+1 yk = sk) while preserving symmetry and, under appropriate conditions, positive definiteness.

Two prominent update formulas are the DFP (Davidon-Fletcher-Powell) update and the BFGS (Broyden-Fletcher-Goldfarb-Shanno) update; the latter is typically favored for numerical stability. For large-scale problems, limited-memory variants (L-BFGS) store only a few vectors to approximate the inverse Hessian, reducing memory and computational demands.

Quasi-Newton methods often exhibit superlinear convergence and are widely used in scientific computing and machine learning due to their robustness and efficiency, particularly when Hessian evaluation is costly. They may be complemented by SR1 updates or trust-region techniques in cases with challenging curvature.