leastsquares

The least squares method is a standard approach for estimating unknown parameters by minimizing the sum of squared deviations between observed values and those predicted by a model. It is widely used in data fitting, regression analysis, and numerical approximation. Historical note: associated with Adrien-Marie Legendre (1805) and Carl Friedrich Gauss; used to fit polynomials.

Suppose y ∈ R^n, X ∈ R^{n×p}, model y ≈ Xβ, with β ∈ R^p. The ordinary least squares estimate minimizes ||y - Xβ||^2. If X has full column rank, the solution β̂ = (X^T X)^{-1} X^T y. Predicted values ŷ = Xβ̂, residuals r = y - ŷ.

Variants include weighted and generalized formulations. In weighted least squares with a positive definite weight matrix W, β̂ = (X^T W X)^{-1} X^T W y. Generalized least squares accounts for correlated errors with covariance Ω, giving β̂ = (X^T Ω^{-1} X)^{-1} X^T Ω^{-1} y.

Nonlinear least squares extend the idea to models where the predictions are nonlinear in the parameters. One minimizes the sum of squares of residuals r_i(β) = y_i - f(x_i, β). Algorithms commonly used include Gauss-Newton and Levenberg–Marquardt, which iteratively adjust β and require sensible initial estimates and convergence criteria.

Properties and scope include assumptions about error structure, such as uncorrelated and homoscedastic errors, with normality enabling straightforward inference in many cases. The conditioning of X affects numerical stability, and sensitivity to outliers motivates robust variants. Applications span econometrics, physics, engineering, computer vision, and signal processing.