Selftraining

Self-training is a semi-supervised learning approach in which a model trained on a small labeled dataset is used to assign labels to unlabeled data, and the model is retrained on the combined labeled and pseudo-labeled data in an iterative loop.

The typical workflow is: train an initial classifier on the labeled data, apply it to the unlabeled pool, select high-confidence predictions using a threshold or top-k, add these pseudo-labeled instances to the training set, and retrain. This process is repeated until convergence or until no new high-confidence samples are found. Confidence estimates can be based on probability scores, margins, or calibration.

Self-training is related to pseudo-labeling and is a simple baseline for semi-supervised learning. It differs from multi-view methods such as co-training or tri-training, which rely on multiple classifiers or feature perspectives. Variants include self-paced learning and approaches that downweight or filter uncertain labels to reduce noise.

Applications span natural language processing, computer vision, speech recognition, and other domains where labeled data is scarce but unlabeled data is abundant. Benefits include leveraging large unlabeled datasets with minimal changes to training pipelines; drawbacks include the risk of propagating errors if the initial model or confidence estimates are biased. Mitigation strategies include careful thresholding, confidence calibration, early stopping, and balancing labeled and pseudo-labeled data.

Theoretical results suggest self-training can be consistent under certain assumptions, such as low-density separation and the cluster assumption, but guarantees are limited and performance depends on data representativeness and reliable confidence estimates.