Regularized

Regularized is an adjective used in statistics, machine learning, and optimization to describe methods that incorporate additional information or constraints to stabilize solutions, prevent overfitting, or handle ill-posed problems. This is typically achieved by modifying an objective function to include a penalty term that discourages excessive model complexity or extreme parameter values.

In practice, regularization adds a penalty to the loss or error that the model seeks to minimize. Common penalties include L2 regularization (ridge), which adds a term proportional to the squared magnitude of the parameter vector, and L1 regularization (lasso), which adds a term proportional to the sum of absolute values of the parameters. Elastic net combines both L1 and L2 penalties. In linear models, L2 regularization is closely related to Tikhonov regularization, a classical technique for stabilizing solutions in inverse problems. Regularization is also used in logistic regression, neural networks, and many other learning algorithms.

From a Bayesian perspective, regularization corresponds to imposing a prior distribution on the model parameters. A Gaussian prior yields an L2 penalty, while a Laplace prior yields an L1 penalty, tying the concept to probability theory.

Choosing the regularization strength, often denoted lambda, controls the bias-variance trade-off: higher regularization tends to simpler, more biased models but can reduce variance and improve generalization. The regularization path—solutions as lambda changes—can be explored with cross-validation to select an appropriate level. Regularization is a fundamental tool for improving robustness and interpretability across predictive modeling and optimization tasks.