RNNt

RNNT, short for Recurrent Neural Network Transducer, is an end-to-end architecture for automatic speech recognition designed to produce transcriptions in streaming fashion. It maps sequences of acoustic features to text tokens while generating outputs incrementally as audio is received, enabling online transcription without requiring full-sequence wait times.

The architecture comprises three main components. The transcription (encoder) network processes the input acoustic features to produce high-level representations. The prediction (decoder) network conditions on previously emitted non-blank symbols to capture context and language information. The joint network combines the encoder and predictor representations to produce a probability distribution over the next symbol, including a special blank token used to model timing and alignments. Training uses the RNNT loss, a dynamic programming objective that sums over all possible alignments between input frames and output tokens, making the model robust to varying lengths and alignments.

During inference, RNNT can operate in a streaming mode, emitting tokens as sufficient evidence accumulates. Decoding typically uses beam search or greedy strategies to find the most likely token sequence. In practice, RNNT models are built with convolutional or conformer encoders and recurrent or transformer-based predictors, emphasizing efficient, low-latency decoding suitable for live speech recognition.

History and context: the RNN Transducer concept originated in early work on neural sequence transduction, with RNNT variants refined by researchers in industry and academia to improve accuracy and efficiency on large-scale speech data. Compared with CTC-based models, RNNT incorporates an explicit predictor network and a joint network, enabling richer output dependencies while maintaining end-to-end trainability and streaming capability.