MCCV

Monte Carlo cross-validation (MCCV) is a resampling method used to estimate the predictive performance of a statistical model. In MCCV, the available data are randomly partitioned into training and validation sets many times. For each iteration, the model is trained on the training subset and evaluated on the validation subset using a chosen performance metric, such as accuracy, AUC, or RMSE. The process is repeated for a user-specified number of iterations, and the reported performance is typically the average of the validation scores across iterations, often accompanied by a standard error or confidence interval.

Procedure and options. In each iteration, a random split is drawn according to a predefined training proportion (for example, 70% training and 30% validation). The splits can be performed with or without stratification to preserve the distribution of the target variable. The number of iterations (B) is chosen in advance and may range from tens to thousands, depending on the dataset size and computational resources. MCCV allows using different training set sizes to assess how performance changes with the amount of available data.

Relation to other cross-validation methods. MCCV is related to but distinct from k-fold cross-validation. Unlike k-fold, MCCV uses random splits that can vary across iterations and does not require that every observation appears in a validation set exactly once. It is also referred to as random-subsampling cross-validation or repeated random sub-sampling validation.

Advantages and limitations. MCCV is flexible and can be advantageous for small datasets or when you want to explore different training-set sizes. However, it can be computationally intensive and may yield estimates with higher variance if not enough iterations are performed. Proper use often involves stratification and reporting both the mean performance and its uncertainty.