DecoderOnly

Decoder-only refers to a class of neural network language models that use solely the decoder component of the Transformer architecture in an autoregressive, left-to-right manner. In these models, each token is generated conditioned on all previous tokens, with causal masking ensuring that future information cannot influence earlier predictions. This contrasts with encoder-decoder models, which separate input processing (encoder) from output generation (decoder).

Training and objective are centered on next-token prediction. Decoder-only models are trained to maximize the likelihood of the next token given the preceding text, using large corpora from diverse sources. After pretraining, they may be fine-tuned for specific tasks, instruction-tuned, or aligned with human preferences through reinforcement learning from human feedback (RLHF). The resulting systems are capable of free-form text generation, code completion, translation in some settings, and conversational dialogue.

Key characteristics include left-to-right generation, the ability to perform zero-shot and few-shot tasks through prompt design, and scalability with increasing parameter counts. Their relatively simple architecture often yields faster inference for long outputs and straightforward deployment compared to encoder-decoder setups. However, they can struggle with tasks that benefit from explicit cross-attention to structured inputs, may require substantial compute and memory, and are prone to hallucinations or biased outputs if not carefully mitigated.

Notable examples of decoder-only models include large language models such as GPT-series and other autoregressive transformers used for chat, writing assistance, and code generation. They remain a common baseline and benchmark in language modeling research due to their strength in generating coherent, contextually relevant text.