encoderdecodertransformere

An encoder–decoder transformer, also called a seq2seq transformer, is a neural network architecture designed for transforming one sequence into another. It uses self-attention to model dependencies within the input and output sequences and cross-attention to relate the two, enabling powerful sequence transduction without recurrent networks.

The architecture consists of two main components. The encoder is a stack of layers, each with a multi-head self-attention sublayer and a position-wise feed-forward network, together with residual connections and layer normalization. The decoder is a stack of layers that include masked multi-head self-attention, an encoder–decoder attention sublayer that attends to the encoder outputs, and a feed-forward network. Input and target sequences are embedded and augmented with positional encodings before processing. The encoder produces a sequence of contextualized representations, which the decoder attends to while generating the output autoregressively.

Training typically uses supervised learning to maximize the likelihood of the target sequence given the input, commonly with cross-entropy loss in a teacher-forcing setup. Inference generates tokens one by one, often using greedy decoding, beam search, or other sampling methods. The attention mechanisms support flexible alignment between input and output tokens without explicit alignment procedures.

Variants and applications: The encoder–decoder transformer underpins many tasks, notably machine translation and text summarization, as well as question answering and multimodal problems. It has inspired variants such as pre-trained encoder–decoder models (e.g., T5, BART) and, in related families, decoder-only and encoder-only architectures (e.g., GPT, BERT) that reuse attention mechanisms.

History and impact: Introduced by Vaswani et al. in 2017, the architecture popularized attention-based sequence modeling and has driven major advances in natural language processing, with broad adoption in research and industry. Limitations include computational and memory intensity on long sequences, prompting research into efficient attention and longer-context handling.