underparameterization

Underparameterization is a concept in machine learning referring to models with fewer parameters than the dimensionality of the input data. In traditional statistical learning theory, such models are expected to have high bias and low variance, leading to underfitting. This means the model is too simple to capture the underlying patterns in the data. For example, trying to fit a linear model to highly non-linear data would be an instance of underparameterization.

However, recent research, particularly in deep learning, has shown that underparameterized models can sometimes exhibit surprising generalization capabilities, especially when trained for a very long time or with specific optimization techniques. This phenomenon, often referred to as the "double descent" or "benign overfitting" effect, suggests that the relationship between model complexity, data size, and generalization is more nuanced than previously understood. In these scenarios, as a model becomes more complex and even goes beyond the point of interpolation (fitting all training data perfectly), its test error can decrease. The exact mechanisms behind this observation are still an active area of research.