overfitting

Overfitting is a modeling error in statistics and machine learning where a model captures not only the underlying patterns in the training data but also random noise. This leads to excellent performance on the training data but poor generalization to new, unseen data.

Causes include high model complexity relative to the amount of data, noisy data, insufficient data, excessive feature engineering, and leakage of information from the training data into the validation or test sets. Highly flexible models such as deep neural networks or complex decision trees can be especially prone to overfitting when data are limited or training continues for too long.

Effects of overfitting include a low training error paired with a high error on unseen data, indicating poor generalization. The model may produce unstable predictions and be sensitive to small changes in input that do not reflect real-world variation.

Detection methods involve monitoring performance on held-out validation data or through cross-validation. A large gap between training and validation error, or validation error that increases during training, signals overfitting. Learning curves and diagnostic tests help assess whether the model will generalize.

Prevention and remedies focus on reducing model variance and improving generalization. Strategies include using simpler or regularized models (L1 or L2 regularization), early stopping, obtaining more data, feature selection or dimensionality reduction, and preventing data leakage. Other techniques such as cross-validation, bagging, and ensembling can reduce variance. In neural networks, dropout, data augmentation, and careful architectural choices are common.

Relation to bias and variance: Overfitting reflects high variance in the model, where capacity exceeds the information available in the data. Balancing bias and variance through appropriate complexity and data handling is essential for robust generalization.