Hyperparameters

Hyperparameters are configuration settings for a machine learning model that are not learned from the data during training. They are set before the learning process begins and govern aspects such as model capacity, learning dynamics, and data processing. Hyperparameters differ from model parameters, which are the weights and biases learned by optimization algorithms to minimize a loss function. Common examples include the learning rate, batch size, regularization strength (L1 or L2), the number of layers and units per layer in a neural network, activation functions, dropout rate, and the choice of optimization algorithm. Other examples include weight initialization method, gradient clipping, data augmentation parameters, and early stopping criteria. Some hyperparameters are global, while others are layer-specific.

Properly chosen hyperparameters can improve convergence speed and generalization, while poor choices can lead to slow training, unstable optimization, underfitting, or overfitting. The learning rate controls step size and is often the most sensitive parameter. Regularization terms constrain model complexity. Architectural choices determine capacity and inductive bias. Hyperparameters are typically set using a validation set or cross-validation. Tuning methods include manual search, grid search, random search, and more sophisticated approaches such as Bayesian optimization, Hyperband and evolutionary strategies. In practice, practitioners balance search thoroughness with computing cost, sometimes using warm starts, historical results, or multi-fidelity evaluations to prune unpromising configurations.

Best practices include starting from reasonable defaults, documenting all choices, and keeping a separation between hyperparameters and architectural decisions. Some hyperparameters are dataset- and model-specific, and meaningful defaults may vary across domains. Automated hyperparameter optimization is increasingly common, particularly for large neural networks, where the search space can be high-dimensional. Reporting the tuned values and the performance outcomes aids reproducibility and comparability across studies.