Codistillation

Codistillation is a learning paradigm in machine learning in which multiple models are trained in concert and exchange information during training, typically by sharing softened output distributions rather than hard labels. In codistillation, each model acts as both student and teacher to the others, using the peers' predictions as auxiliary targets in addition to ground-truth labels. This approach is commonly used in semi-supervised learning, ensemble methods, and scenarios with limited labeled data.

Typically, training involves a combination of a standard supervised loss on labeled data and a distillation loss that measures divergence between a model's output and the soft labels produced by the other models on the same data or on unlabeled data. Temperature scaling is often employed to soften probabilities and reveal information about model uncertainty. Variants include mutual distillation, cross-teaching, and peer-prediction schemes, where models are encouraged to converge to a consensus while preserving diversity.

Benefits include improved accuracy and calibration when labeled data are scarce, better utilization of unlabeled data, and robustness to label noise. Limitations include higher computational cost due to training several models, sensitivity to hyperparameters, and potential convergence to similar error modes if models are not sufficiently diverse. Codistillation is related to knowledge distillation but emphasizes collaborative, multi-model learning rather than a one-way transfer from a single teacher.

See also: knowledge distillation, semi-supervised learning, co-training, ensemble learning.