nonautoregressive

Nonautoregressive refers to models that generate output tokens in parallel rather than sequentially, conditioning on all previously generated content in real time. In contrast, autoregressive models produce tokens one at a time, each conditioned on the entire history of earlier outputs. Nonautoregressive (NAR) approaches aim to speed up sequence generation by reducing or eliminating the dependency on prior outputs during decoding. They are widely studied in natural language processing, especially in neural machine translation, but have also appeared in speech and text generation tasks.

Fully nonautoregressive methods attempt to produce the entire target sequence in a single step. A common challenge for these models is accurately modeling dependencies between target tokens, which can lead to lower fluency or coherence compared with autoregressive counterparts. To address this, several strategies have been proposed. Iterative refinement methods generate an initial, parallel prediction and progressively refine it through multiple rounds. Examples include Mask-Predict and variants that apply controllable edits to an initial sequence. Fertility-based approaches attempt to predict the number of target tokens for each source segment, enabling more structured parallel generation. Distillation from autoregressive teachers is frequently used to stabilize training.

Advantages of nonautoregressive models include reduced inference latency and better parallelism on modern hardware, making them attractive for real-time or large-scale generation. Limitations often involve a trade-off between speed and output quality, difficulties in capturing long-range dependencies, and sensitivity to modeling choices and training objectives.

Applications commonly focus on machine translation and other text generation tasks where fast decoding is beneficial. Ongoing research seeks to close the quality gap with autoregressive methods while preserving the efficiency gains of parallel generation.