Valset

Valset, short for validation set, is a subset of a dataset reserved for assessing a model during its development. It is distinct from the training set and is used to gauge how well the model generalizes to new data without touching the final test data. The valset should reflect the same distribution as the training data to provide a meaningful estimate of performance on unseen examples.

In practice, a portion of labeled data is allocated to the valset, commonly in the range of a few percent up to about 20% depending on the total data size. Stratified sampling is often used for classification tasks to preserve class proportions. It is important to avoid leakage, meaning no information from the training process or data augmentation applied to training should influence the valset.

The valset serves several roles during development. It is used to monitor metrics such as loss or accuracy during training, to detect overfitting, and to guide hyperparameter tuning. Early stopping may rely on validation performance to halt training when improvements cease. After model selection, a separate test set is typically used for the final, unbiased evaluation of performance.

Variations exist, including cross-validation, where multiple valsets are created by partitioning the data into train and validation folds to obtain more robust performance estimates. In time series contexts, validation must respect temporal order to avoid leakage from future data.

Many machine learning frameworks support separate validation data by providing data loaders or train-validation splits, with valset processing often excluding data augmentation and maintaining consistent preprocessing.