Regularizers

Regularizers are techniques added to the loss function during training to discourage model complexity and prevent overfitting. By penalizing large or numerous weights, regularizers constrain the capacity of a model, encouraging simpler, more generalizable solutions.

Common forms include L1 regularization (lasso) and L2 regularization (ridge). L1 adds a penalty proportional to the sum of absolute weights, promoting sparsity by driving some weights to zero. L2 adds a penalty proportional to the sum of squared weights, shrinking weights toward zero without necessarily eliminating features. Elastic Net combines L1 and L2 penalties to balance sparsity and stability, useful with correlated features.

Weight decay is a closely related concept often implemented as L2 regularization in optimization algorithms. Other training-time approaches with regularizing effects include dropout, which randomly deactivates units during training to reduce co-adaptation, and early stopping, which halts training once performance on a validation set stops improving.

Data augmentation can act as a regularizer by increasing input diversity and robustness. Max-norm constraints limit the magnitude of weight vectors to prevent large updates. Some methods interpret regularization from a Bayesian perspective: L2 corresponds to a Gaussian prior on weights, and L1 to a Laplace prior, framing regularization as incorporating prior beliefs about parameter size.

Regularizers are widely used across supervised learning tasks, including linear models, logistic regression, and neural networks. The strength of the regularization is a hyperparameter often selected by cross-validation. Excessive regularization can cause underfitting, while insufficient regularization may fail to mitigate overfitting.