Hessianapproximationen

Hessian approximations refer to techniques used to estimate or replace the true Hessian matrix, which contains second-order partial derivatives of a scalar-valued function with respect to its variables. Directly computing the Hessian can be computationally expensive or infeasible for high-dimensional problems, so approximations are employed in optimization algorithms to balance efficiency and accuracy.

In unconstrained nonlinear optimization, second-order methods such as Newton’s method require the exact Hessian to determine the search direction. When the Hessian is unavailable or costly, quasi‑Newton algorithms build an approximate Hessian iteratively. Methods such as BFGS (Broyden–Fletcher–Goldfarb–Shanno) and its limited‑memory variant L‑BFGS update an approximate inverse Hessian using gradient differences and step vectors, ensuring positive definiteness and superlinear convergence under mild assumptions.

Other Hessian approximation strategies include finite difference approaches, where second derivatives are approximated by evaluating gradients at nearby points, and diagonal or block‑diagonal models that reduce storage requirements. Randomized Hessian sketches and subsampling techniques are also employed in stochastic optimization to handle extremely large datasets.

Hessian approximations are crucial in fields ranging from machine learning to engineering design, where large-scale optimization problems are common. By using these approximations, algorithms achieve improved convergence rates compared to first‑order methods while keeping computational resources manageable.