Vits

VITS, short for Variational Inference TTS, is a neural text-to-speech framework designed to generate natural-sounding speech in an end-to-end fashion. It aims to simplify traditional TTS pipelines by learning a direct mapping from text to waveform or to high-quality intermediate representations within a single model, using variational inference and adversarial training to improve realism and stability.

The core idea of VITS is to model the acoustic front end with a variational latent representation, capturing the variability in speech such as pitch, timing, and timbre. The model typically combines an encoder that processes linguistic or phonetic features, a stochastic latent variable module with a variational posterior and a prior, and a decoder that produces acoustic features and a neural vocoder component to synthesize waveform. A normalizing flow is employed to flexibly transform latent distributions, enabling the model to better fit the observed data. Adversarial losses and auxiliary objectives are used to sharpen realism of the generated audio and improve prosody.

VITS supports multi-speaker synthesis and can be conditioned on speaker identity or embedding to control voice characteristics. It is trained on paired text and audio data and is designed to produce high-quality speech without the need for separate duration models or a separately trained vocoder in a traditional pipeline. The approach has been influential for its end-to-end simplicity and competitive naturalness, and several open-source implementations have been released by researchers and practitioners in the field.

Limitations include the need for substantial training data and computational resources, as well as potential sensitivity to dataset biases or language-specific prosody. VITS remains part of ongoing work to improve end-to-end speech synthesis and cross-language applications.

See also: text-to-speech, variational autoencoder, normalizing flow, neural vocoder.