Pretraining

Pretraining refers to training a model on a broad dataset to learn general representations that can be adapted to many downstream tasks. After pretraining, the model is fine-tuned on a task with task-specific labeled data. This approach is widely used in natural language processing, computer vision, and multimodal learning because it can improve performance and reduce the need for large labeled datasets.

In natural language processing, pretraining commonly uses self-supervised objectives such as masked language modeling or autoregressive language modeling. Additional tasks may include predicting sentence order or other proxy objectives. In computer vision, pretraining may involve self-supervised or supervised objectives, including contrastive learning on large image collections. The pretrained representations are later fine-tuned for tasks such as classification, question answering, translation, or generation.

Pretrained models are typically trained on very large corpora to capture broad structural information. In NLP, data sources include web text, books, and encyclopedic content; in vision, diverse image collections are used. Training at this scale requires substantial compute and memory, and data quality and biases in the data can influence model behavior and fairness.

Fine-tuning adapts the pretrained representations to a specific task with labeled data, and practitioners may perform domain-adaptive pretraining on task-relevant data before fine-tuning. Pretraining reduces data requirements for downstream tasks and can improve transfer performance, but effectiveness depends on the similarity between pretraining data and the target domain, as well as model size and training regimen.

Variants include supervised pretraining on labeled data, multimodal pretraining that combines text and images, and continual or lifelong pretraining. Evaluation focuses on transfer learning performance across benchmarks, robustness, and efficiency in real-world applications.