BiasVariance

Bias-variance is a concept in statistics and machine learning describing two sources of error in predictive modeling. Bias refers to systematic error introduced by approximating a real-world problem with a simplified model. Variance refers to the model’s sensitivity to fluctuations in the training data, causing predictions to vary across different training sets. Together with irreducible noise, these components determine the expected generalization error.

For a given input x, and under squared error loss, the expected squared prediction error over different training sets can be decomposed as: E[(Y - f_hat(x))^2] = Bias^2(f_hat, x) + Var(f_hat, x) + σ^2(x), where f_hat is the learned predictor, Bias(f_hat, x) = E[f_hat(x)] - f(x) is the bias, Var(f_hat, x) = E[(f_hat(x) - E[f_hat(x)])^2] is the variance, and σ^2(x) = Var(Y|X=x) is the irreducible noise. This decomposition shows the trade-off: increasing model complexity often reduces bias but increases variance, while a simpler model has lower variance but higher bias.

Practical implications include choosing models and tuning methods to balance the trade-off. Regularization, cross-validation, and model selection help manage bias and variance. Ensemble methods such as bagging primarily reduce variance, while boosting aims to reduce bias. The optimal balance depends on data size, noise level, and the learning task.