encoderonly

Encoder-only models are transformer-based neural networks that focus on encoding input sequences into contextualized representations without producing autoregressive outputs. They are designed for understanding and representation tasks rather than text generation, distinguishing them from decoder-only and encoder-decoder architectures.

These models typically consist of multiple stacked transformer encoder layers that apply self-attention to the input and feed-forward networks to transform token representations. They generate rich token-level embeddings and often a pooled representation for entire sequences, which can be used for classification, similarity, or retrieval tasks. Pretraining commonly uses masked language modeling, where a portion of tokens are masked and the model predicts them. Some variants also employ additional objectives such as sentence-order prediction or more discriminative pretraining tasks (as in ELECTRA). After pretraining, the models are finetuned on specific downstream tasks with labeled data.

Common examples of encoder-only models include BERT, RoBERTa, ALBERT, ELECTRA, DistilBERT, and domain-specific variants like BioBERT. They have become standard tools for a wide range of natural language processing tasks, including text classification, named entity recognition, sentiment analysis, question answering when framed as embedding or classification tasks, and information retrieval where high-quality representations are valuable.

Compared with decoder-only models, encoder-only architectures excel at understanding and embedding text rather than generating it. They differ from encoder-decoder models, which pair encoding with a separate decoding component for generation. Limitations of encoder-only models include their focus on representation rather than fluent generation, substantial computational requirements for large variants, and potential biases present in training data. Overall, they remain central to many language understanding applications.