Regularisation

Regularisation, in statistics, machine learning, and numerical analysis, refers to techniques that introduce additional information to constrain a model or ill-posed problem. The goal is to prevent overfitting, improve generalization to unseen data, and stabilize computations by penalizing extreme parameter values or overly complex solutions.

In supervised learning, regularisation is commonly implemented by adding a penalty term to the loss function. L1 regularisation (lasso) uses the L1 norm of the parameter vector and tends to produce sparse models. L2 regularisation (ridge) uses the squared L2 norm and shrinks coefficients toward zero without zeroing them. Elastic net combines both penalties. There are other forms such as group lasso and total variation depending on structure. The penalty strength is controlled by a hyperparameter (often denoted lambda or C).

Bayesian interpretation: regularisation can be viewed as imposing a prior distribution on model parameters; a Gaussian prior yields ridge-like penalties, a Laplace prior yields lasso-like penalties. The regularisation strength corresponds to the prior precision or scale.

In inverse problems and numerical analysis, regularisation (such as Tikhonov regularization) stabilizes solutions to ill-conditioned problems by discouraging large or rough solutions, often through smoothness or norm constraints.

Trade-offs: regularisation reduces variance and improves generalization but introduces bias and can complicate interpretation. It aids with multicollinearity and feature selection (L1). Selection of the regularisation strength is typically done via cross-validation or information criteria.