WaveNet

WaveNet is a deep neural network architecture for generating raw audio waveforms. Introduced by DeepMind in 2016, it is a probabilistic generative model that defines the conditional distribution of each audio sample given all previously generated samples, often conditioned on auxiliary features such as text transcripts, phonetic representations, or speaker identity.

At its core, WaveNet uses a stack of dilated causal convolution layers. The dilations allow the model to have a very large receptive field with relatively few layers, enabling it to capture long-range temporal dependencies in audio. Each layer feeds through residual and skip connections, and the final output is produced by modeling the conditional distribution of a quantized audio sample (often using mu-law quantization) via a softmax or mixture distribution.

Training is performed by maximum likelihood on large speech or audio datasets, with conditioning signals supplied during both training and synthesis. When used as a speech vocoder, WaveNet converts a time-aligned acoustic representation (such as a mel-spectrogram) into a high-fidelity waveform, generating more natural prosody and articulation than traditional concatenative or parametric methods.

Limitations include slow sample-by-sample generation, which hindered real-time use. Subsequent work introduced means to synthesize in parallel, notably Parallel WaveNet, which uses distillation to train a non-autoregressive model that can generate samples in parallel, and related architectures like WaveRNN.

WaveNet significantly influenced neural text-to-speech and audio synthesis, reinforcing the viability of neural vocoders and enabling more natural-sounding speech in commercial systems. It is widely cited as a foundational model in the field.