ridgeregression

Ridge regression is a linear regression technique that adds L2 regularization to address overfitting and multicollinearity. It minimizes the objective function the sum of squared residuals plus a penalty proportional to the squared magnitude of the coefficient vector: min_beta ||y - X beta||^2 + lambda ||beta||^2, where lambda >= 0 is the regularization strength. The intercept is typically treated separately, and features are usually standardized before fitting to ensure the penalty affects all coefficients comparably.

The solution is closed form: beta_hat = (X^T X + lambda I)^{-1} X^T y (with I the identity). When the intercept is included, a common approach is to center X and y so the intercept is absorbed; or to exclude the intercept from the penalty. Increasing lambda increases bias but reduces variance, shrinking coefficients toward zero; as lambda approaches zero, ridge regression converges to ordinary least squares. For very large lambda, coefficients shrink toward zero.

Ridge regression improves conditioning of the problem, stabilizing estimates in the presence of multicollinearity and highly correlated predictors. It can be interpreted as a Bayesian model with a Gaussian prior on beta; it also relates to PCA by dampening components with small explained variance. Unlike Lasso (L1), ridge regression does not force coefficients to be exactly zero, so it does not perform feature selection by itself.

Practical use involves selecting lambda by cross-validation, information criteria, or Bayesian methods. It is commonly implemented in statistics and machine learning libraries and can be extended to generalized linear models and kernel methods (kernel ridge regression) for non-linear relationships. When using ridge, standardization of features and appropriate handling of the intercept are important for meaningful results.