selfsupervision

Self-supervision is a form of representation learning in which a model learns from unlabeled data by solving automatically generated tasks, called pretext tasks. The supervisory signal is derived from the data itself, not human annotations. The learned representations can be transferred to downstream tasks with limited labeled data, reducing labeling requirements compared to fully supervised learning.

Pretext tasks can be predictive, reconstructive, or contrastive. Predictive tasks require the model to infer missing parts or future content; reconstructive tasks attempt to restore a missing portion; contrastive methods train the model to bring representations of augmented views of the same example closer while pushing apart others.

Notable self-supervised approaches in vision include colorization, inpainting, rotation prediction, and jigsaw puzzles; in natural language processing, masked language modeling and next sentence prediction. Contrastive learning methods such as SimCLR, MoCo, BYOL, and SwAV have achieved strong results, often using large-scale data and carefully designed augmentations.

Evaluation typically uses linear evaluation or fine-tuning on downstream tasks. Large unlabeled datasets are common, and representation quality depends on data augmentation, architecture, and optimization. Potential challenges include reliance on task-specific pretext signals, risk of representation collapse, and computational demands.

Applications include computer vision, natural language processing, speech, and time-series analysis. Self-supervision remains an active research area with ongoing work to understand why certain pretext tasks generalize and how to design tasks that yield robust, transferable representations.