WGAN

The Wasserstein Generative Adversarial Network, or WGAN, is a type of generative adversarial network (GAN) designed to improve the training stability of GANs. Traditional GANs suffer from issues like vanishing gradients and mode collapse, making them difficult to train effectively. WGAN addresses these problems by using the Wasserstein distance, also known as the Earth Mover's distance, as its loss function instead of the Jensen-Shannon divergence or Kullback-Leibler divergence commonly used in earlier GAN architectures.

The Wasserstein distance provides a more meaningful measure of the distance between two probability distributions, even when they have non-overlapping support. This is crucial for GAN training because the generator and discriminator are essentially trying to match the distribution of generated data to the distribution of real data. By employing the Wasserstein distance, WGAN offers a smoother gradient that is less prone to vanishing, leading to more stable training.

To implement the Wasserstein distance, WGAN requires the discriminator, often referred to as the critic in this context, to be trained to approximate the Kantorovich-Rubinstein duality. This involves removing the sigmoid activation from the output layer of the critic and enforcing a Lipschitz constraint on its weights. The Lipschitz constraint ensures that the gradients of the critic remain bounded, which is essential for the Wasserstein distance to be well-defined and for the gradients to be informative during training.

The benefits of WGAN include improved training stability, a reduced likelihood of mode collapse, and a loss metric that correlates with the quality of generated samples. This correlation means that the critic's loss can serve as a reliable indicator of how well the generator is performing, unlike the loss in standard GANs which can be misleading. WGAN and its subsequent variations have significantly advanced the field of GAN research and application.