Regression

Regression is a statistical method for modeling the relationship between a dependent variable and one or more independent variables. It is used for prediction, estimation of conditional means, and assessing associations. In simple linear regression there is a single predictor and a straight-line relationship; in multiple regression several predictors explain the outcome.

Common forms include linear regression, polynomial regression, and generalized linear models, which extend regression to non-normal outcomes such as binary or count data. Nonlinear regression handles curved relationships. Regularization techniques such as ridge, lasso, and elastic net add penalties to improve prediction and reduce overfitting when many predictors or multicollinearity are present.

Estimation typically proceeds by least squares for linear models or by maximum likelihood in generalized models. The Gauss–Markov theorem asserts that, under standard assumptions (linearity in parameters, independent errors with constant variance, and no perfect multicollinearity), the ordinary least squares estimators have desirable properties. Inference relies on assumptions about error distribution, often normality.

Model evaluation uses metrics such as R-squared, adjusted R-squared, root mean squared error (RMSE), and mean absolute error (MAE); cross-validation is common to assess predictive performance. Diagnostics include residual plots, tests for multicollinearity, heteroscedasticity, and influential observations.

Regression has a long history dating to methods developed by Legendre and Gauss for fitting lines to data. It remains a foundational tool in statistics, econometrics, and data science, applied across disciplines for prediction, hypothesis testing, and causal analysis when appropriate assumptions hold.