crossinvalidation

Crossinvalidation is a resampling validation strategy used in statistics and machine learning to assess how well a model generalizes across different data sources or conditions. Unlike standard cross-validation, which randomizes and partitions samples within a single dataset, crossinvalidation explicitly tests transferability across sources, batches, studies, or domains. It can help reveal model performance under distribution shifts and detect overfitting to idiosyncrasies of a particular dataset.

In its simplest form, leave-one-source-out crossinvalidation: data are grouped into sources; in each fold, train on all sources except one, test on the held-out source. Hyperparameter tuning should be performed within the training sources to avoid leakage; nested crossinvalidation can be used if hyperparameter optimization is necessary. More generally, crossinvalidation can be implemented across multiple levels of sources or domains.

Use cases include multi-study clinical research, genomics with batch effects, federated learning setups, and domain generalization experiments where the aim is to quantify performance under distributional shifts. It is particularly relevant when data come from heterogeneous sources or are collected under different conditions, and the goal is to estimate robustness of models to such variations.

Advantages include improved estimation of cross-domain generalization and reduced risk of optimistic bias due to dataset-specific patterns. Limitations include higher variance when there are few sources, greater computational cost, and the need for clearly defined sources to avoid leakage. Related concepts include cross-validation, external validation, leave-one-group-out validation, and domain adaptation.