underparameterized

Underparameterized refers to a model that has fewer parameters than the amount of training data. In machine learning, this often suggests that the model is too simple to capture the underlying complexity of the data. An underparameterized model may struggle to generalize well to new, unseen data, leading to high bias. This means the model makes strong assumptions about the data that might not be true.

In contrast, an overparameterized model has significantly more parameters than training data points. While overparameterization was once thought to be problematic due to the risk of overfitting, modern deep learning models often thrive in such regimes. They can achieve very low training error and, surprisingly, can still generalize well.

The concept of underparameterization is important for understanding model capacity and the bias-variance tradeoff. A model with too few parameters (underparameterized) is likely to have high bias and low variance. Conversely, a model with too many parameters (overparameterized) could potentially have low bias but high variance, leading to overfitting if not managed properly.

Research has explored the behavior of underparameterized models in specific contexts, such as linear regression with more features than samples, where solutions might not be unique. Understanding whether a model is underparameterized or overparameterized helps in diagnosing performance issues and selecting appropriate model architectures and regularization techniques.