regularizer

A regularizer, in statistics and machine learning, is a component added to a loss function to constrain model complexity and improve generalization. Formally, with a loss L(θ) on training data and a parameter vector θ, a regularized objective takes the form L(θ) + λR(θ), where R is a penalty function and λ ≥ 0 is the regularization strength. By increasing λ, the model is discouraged from adopting large or complex parameter values.

Common regularizers include L2 (R(θ) = ||θ||2^2), also called ridge, which tends to shrink weights toward zero without forcing many to zero; L1 (R(θ) = ||θ||1), or lasso, which can produce sparse solutions by setting some coefficients exactly to zero; and Elastic Net, which combines L1 and L2 penalties. Other forms include Group Lasso, Fused Lasso, and nuclear norm penalties for matrices. The choice of regularizer reflects desired properties such as sparsity, smoothness, or group structure.

From a Bayesian perspective, regularization corresponds to placing priors on the parameters. For example, a Gaussian prior yields L2 regularization, while a Laplace prior yields L1 regularization. This linkage helps interpret regularizers as imposing beliefs about parameter distributions.

Optimization and computation vary by regularizer. L1 is convex but non-differentiable at zero, often handled by proximal methods or coordinate descent; L2 is differentiable and amenable to standard gradient-based optimization. The regularization strength λ is typically chosen via cross-validation or information criteria. In neural networks, weight decay refers to L2 regularization, while other techniques such as dropout and early stopping are regularization strategies that influence training rather than adding a penalty term.