autoencoder

An autoencoder is a type of neural network that learns to compress data into a lower-dimensional latent representation and then reconstruct the original input from that representation. It consists of two parts: an encoder that maps input x to a latent code z, and a decoder that maps z back to a reconstruction x_hat. The model is trained to minimize the difference between x and x_hat.

Training is typically performed with backpropagation using gradient descent. Loss functions include mean squared error for continuous data or cross-entropy for binary data. The latent dimension is usually smaller than the input to force compression (an undercomplete autoencoder); if the latent size is large, the model may not learn meaningful representations unless regularization is applied.

Several variants exist. A denoising autoencoder corrupts the input and learns to reconstruct the original, improving robustness to noise. A sparse autoencoder imposes a sparsity constraint on latent activations. A contractive autoencoder penalizes changes in the encoder output with respect to its input. A variational autoencoder introduces probabilistic latent variables and optimizes a lower bound with a KL divergence term.

Autoencoders are used for dimensionality reduction, feature learning, and data denoising. They serve as a preprocessing step for supervised tasks, act as generative models in some configurations, and are useful for anomaly detection. Convolutional autoencoders are common for images, while recurrent variants handle sequences.

Evaluation focuses on reconstruction accuracy and the usefulness of latent features for downstream tasks. Potential issues include overfitting, overly smooth latent spaces, or trivial solutions in overcomplete settings without regularization. Proper normalization and regularization help ensure meaningful representations.