Regularisierungsmethode

Regularization, in statistics and machine learning, refers to techniques that constrain a model's complexity to improve generalization on unseen data. This is typically achieved by adding a penalty term to the loss function or by imposing constraints on model parameters. The penalty discourages large coefficients, reducing variance at the cost of some bias, which can improve predictive performance when data are noisy or when there are many features.

Common regularization methods include L1 regularization (Lasso), which adds the sum of absolute values of the coefficients; L2 regularization (Ridge), which adds the sum of squares of the coefficients; and Elastic Net, which combines both. In linear and generalized linear models, these penalties shrink coefficients toward zero; L1 can drive some coefficients exactly to zero, yielding feature selection, while L2 tends to shrink coefficients smoothly. Elastic Net is often preferred when features are correlated.

Historically, regularization has roots in Tikhonov regularization in the 1960s and ridge regression introduced by Hoerl and Kennard in 1970. The Lasso was proposed by Tibshirani in 1996, and the Elastic Net by Zou and Hastie in 2005. Beyond linear models, regularization is widely used in neural networks, where weight decay (L2) and dropout act as regularizers to improve generalization.

Practically, selecting the regularization strength is typically done by cross-validation. Data scaling matters because penalties depend on the scale of the features. Regularization is not a substitute for good model specification; it helps manage overfitting but may not overcome underfitting if the model is overly simplistic.