OLS

Ordinary Least Squares (OLS) is a method for estimating the parameters of a linear regression by minimizing the sum of squared residuals. In matrix form, the model is y = Xβ + ε, where y is n×1, X is n×k, β is k×1, and ε is the error term. If X has full column rank, the OLS estimator is β̂ = (X′X)^{-1}X′y.

Gauss–Markov assumptions include linearity, exogeneity (E[ε|X] = 0), no perfect multicollinearity, homoscedasticity (Var(ε|X) = σ^2I), and no autocorrelation. Under these conditions, β̂ is BLUE: the best linear unbiased estimator. If errors are normally distributed, β̂ is normal with mean β and variance σ^2(X′X)^{-1}, enabling t-tests and F-tests for inference.

Interpretation and use: ŷ = Xβ̂ predicts responses, and coefficients reflect marginal effects holding other variables constant. Diagnostics examine residuals for patterns, and tests assess heteroscedasticity or autocorrelation and identify influential observations.

Limitations include sensitivity to outliers, potential endogeneity, and multicollinearity, which inflates the variance of estimates. Heteroscedasticity or autocorrelation can bias standard errors. Alternatives include generalized least squares, weighted least squares, robust standard errors, and regularization methods such as ridge and LASSO. Extensions cover generalized linear models for non-normal outcomes.

OLS is a foundational tool in economics, social sciences, and natural sciences, and is implemented in most statistical software as a baseline model.