Tacotron

Tacotron is a neural network architecture for end-to-end speech synthesis developed by researchers at Google. Introduced in 2017, it aims to convert text input directly into natural-sounding speech by predicting intermediate acoustic representations, typically mel-spectrograms, that are then converted into waveforms by a vocoder.

Tacotron is a sequence-to-sequence model with attention. The encoder converts input text (characters or phonemes) into a high-level representation using a convolutional bank, highway network, and bidirectional recurrent units, a module commonly referred to as CBHG. The decoder is autoregressive and employs attention to align the encoded text with the generated speech frames, producing one mel-spectrogram frame per time step. A postnet, consisting of a stack of convolutional layers, further refines the predicted spectrogram to improve quality.

Training uses paired text-audio data, optimizing a loss that includes the mel-spectrogram error and postnet refinement error. The model outputs a mel-spectrogram, which must be converted to a waveform by a vocoder such as Griffin-Lim or a neural vocoder like WaveNet. In practice, neural vocoders began to play a central role in achieving higher fidelity in later systems.

Tacotron inspired subsequent developments, most notably Tacotron 2, which integrates a revised encoder–decoder with a more robust attention mechanism and uses a neural vocoder (often WaveNet or similar) to synthesize audio, achieving higher naturalness. The architecture has influenced many end-to-end TTS systems and remains a foundational model in the field.

Limitations include reliance on large datasets for training, potential mispronunciations or misalignments, and the computational intensity of training and synthesis.