LeastSquaresFitting

LeastsquaresFitting is the process of estimating the parameters of a model by minimizing the sum of squared residuals between observed data and the model’s predictions. It is a foundational approach in statistics and numerical analysis for fitting curves and regression models to data.

In the common linear form, data are modeled as y ≈ Xβ, where y is a vector of observations, X is a design matrix of predictors, and β is a vector of unknown coefficients. The objective is to minimize the squared error ||y − Xβ||^2. If X has full column rank, the least squares estimate is β̂ = (X^T X)^{-1} X^T y. Numerically stable solutions are often obtained via QR decomposition or singular value decomposition (SVD) rather than formation of the normal equations.

Nonlinear least squares generalizes the approach to models y ≈ f(x, β) where f is nonlinear in β. The residual sum of squares S(β) = ∑ [y_i − f(x_i, β)]^2 is minimized iteratively using methods such as Gauss-Newton or Levenberg–Marquardt.

Variants include weighted least squares, which incorporates measurement uncertainties by minimizing (y − f)^T W (y − f), where W is a weight matrix (often diagonal). Generalized least squares accounts for correlated or non-constant variance errors via a specified covariance matrix.

History and applications: least squares methods trace to Legendre and Gauss in the early 19th century and are now standard in science and engineering for calibration, trend fitting, and statistical regression.

Limitations include sensitivity to model misspecification and outliers; robust or M-estimator variants address these issues. Proper model selection and diagnostic checks are essential to reliable inference.