Pretrain

Pretrain, short for pretraining, refers to the practice of training a machine learning model on a broad, general dataset before adapting it to a specific downstream task. The aim is to learn versatile representations that can be refined with relatively small amounts of task-specific data, improving performance and reducing the need for extensive labeled datasets.

In natural language processing, preprocessing typically uses self-supervised objectives such as masked language modeling or autoregressive language modeling, enabling models to learn linguistic structure from unlabeled text. In computer vision, self-supervised or supervised pretraining teaches general visual features through tasks like contrastive learning, autoencoding, or predictive modeling. Multimodal pretraining combines signals across modalities, for example aligning text and image representations.

After pretraining, the model is fine-tuned on a downstream task, such as classification, translation, or question answering. The combination of pretraining and fine-tuning enables transfer learning: the pretrained weights provide a starting point that often requires less labeled data and training time than training from scratch.

Pretraining is computationally intensive and typically conducted on large corpora or curated datasets. The resulting pretrained models can be released publicly or kept proprietary. Risks include data biases, privacy concerns, and distribution mismatch between pretraining and downstream tasks, which can limit generalization.

Examples include BERT, GPT, and CLIP in their respective domains, as well as many domain-specific language or vision models. The field continues to explore better objectives, data curation, and more efficient fine-tuning.