KvasiNewton

KvasiNewton methods are a class of iterative optimization algorithms used for unconstrained minimization. They aim to efficiently find minima by building low-rank approximations to the Hessian matrix, or its inverse, based on gradient information from successive iterations. The term quasi-Newton denotes updates that incorporate curvature information without requiring exact second-derivative calculations.

At iteration k, with current point x_k and gradient g_k = ∇f(x_k), one defines s_k = x_{k+1} − x_k and y_k = g_{k+1} − g_k. The update produces a new approximation B_{k+1} to the Hessian (or H_{k+1} to the inverse Hessian) that satisfies the secant equation B_{k+1} s_k = y_k and is constructed to be symmetric. Under suitable conditions, the update also preserves positive definiteness, which supports stable, descent-based optimization.

The most widely used variant is BFGS (Broyden–Fletcher–Goldfarb–Shanno), known for robust performance and positive definiteness when y_k^T s_k > 0. Other classical forms include DFP (Davidon–Fletcher–Powell). For large-scale problems, limited-memory variants such as L-BFGS store only a small number of vectors, reducing memory usage while retaining strong convergence properties.

KvasiNewton methods typically achieve fast practical convergence, often superlinear and sometimes near-quadratic, particularly on smooth objectives. They require gradient evaluations per iteration and avoid full Hessian computations, making them well-suited to a wide range of applications in science, engineering, and machine learning.

Limitations include sensitivity to scaling and line-search or trust-region choices, potential difficulties with non-smooth or ill-conditioned problems, and the need for appropriate initialization to ensure robust behavior.