selfsupervised

Self-supervised learning refers to methods that learn useful representations from data without requiring manual labels by constructing tasks whose supervision signals come from the data itself. It sits between supervised learning, which relies on labeled examples, and unsupervised learning, which aims to discover structure without explicit targets. The central idea is to train a model to solve a pretext task, and in the process learn representations that transfer to downstream tasks such as classification or detection.

Approaches include generative, discriminative, and hybrid methods. In vision, pretexts include predicting missing regions, colorization, inpainting, or solving jigsaw puzzles; in NLP, predicting masked tokens or adjacent sentences. Contrastive learning treats augmented views of the same input as positives and different inputs as negatives to align representations. Notable families include SimCLR and MoCo (contrastive) and BYOL and DINO (non-contrastive). Generative approaches include masked modeling (e.g., BERT) and autoencoders; autoregressive models (e.g., GPT) also rely on self-generated targets. Data augmentation plays a crucial role in creating robust representations.

The learned representations are evaluated by training a simple downstream model on labeled data (linear evaluation) or by fine-tuning on the target task. Self-supervised learning has found widespread use in computer vision, natural language processing, and speech, enabling models to leverage large unlabeled corpora and reduce labeling costs while improving transfer performance.

Advantages include scalability and data efficiency; limitations include sensitivity to pretext task design, potential to capture spurious cues, and substantial computational requirements. Ongoing work seeks better pretext tasks, principled evaluation, and reliable transfer across domains.