autoencoders

An autoencoder is a type of neural network that learns a compact representation of input data by training to reconstruct the input from a latent representation. It consists of an encoder, which maps input x to a latent code z, and a decoder, which maps z back to a reconstruction x̂. The network is trained by minimizing a reconstruction loss, such as mean squared error for continuous data or cross-entropy for binary data. The latent code is typically lower-dimensional, enforcing compression and encouraging the model to capture meaningful structure.

Autoencoders can be shallow or deep, and undercomplete (latent dimension smaller than input) to enforce compression, or overcomplete with regularization to prevent trivial solutions. Regularization techniques include sparse autoencoders, which promote sparsity in z; denoising autoencoders, which reconstruct the original input from a corrupted version; and contractive autoencoders, which penalize the sensitivity of the encoder to input changes.

Variational autoencoders (VAEs) add a probabilistic framework, with the encoder producing parameters of a distribution over latent variables and the decoder generating data from samples. Training optimizes a reconstruction term plus a KL-divergence term to regularize the latent distribution, enabling generative capabilities and smooth latent spaces.

Autoencoders are used for dimensionality reduction, feature learning, denoising, anomaly detection, and as building blocks in representation learning. Extensions include convolutional autoencoders for images and recurrent autoencoders for sequences. Limitations include potential to learn identity mappings without proper regularization and the need for careful architectural choices and evaluation.