LOOCV

Leave-One-Out Cross-Validation (LOOCV) is a cross-validation method used to estimate the predictive performance of a statistical model. In LOOCV, each observation in a dataset is used once as a test example, while the remaining n-1 observations form the training set. The process is repeated n times, producing n predicted test values; the LOOCV error is the average of the losses across these held-out observations.

Procedure: Given a dataset with n observations, for i from 1 to n, fit the model on the data excluding observation i, generate a prediction for i, and compute a loss L(y_i, y_hat_i). The LOOCV error is the mean of L over all i. The loss can be the squared error for regression or misclassification/log loss for classification. For some models, particularly linear models with squared error, LOOCV can be computed without refitting by using the hat matrix; otherwise, the straightforward approach requires fitting the model n times, which can be computationally expensive for large n.

Advantages and limitations: LOOCV has a low bias as an estimator of generalization error because almost all data are used for training in each split. However, it often has high variance since the training sets differ by only one observation, and results can be sensitive to outliers. It is most practical for small datasets; for larger datasets, k-fold cross-validation with a modest number of folds is typically preferred. LOOCV is commonly used for model selection and hyperparameter tuning, but should be interpreted with caution when the data are noisy or when robust estimation is important.