LSTMs

Long short-term memory networks, or LSTMs, are a type of recurrent neural network designed to model sequential data. They were introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997 to address the vanishing and exploding gradient problems that can hinder learning long-range dependencies in standard RNNs. LSTMs maintain a memory cell that can preserve information over long time spans, controlled by trainable gates.

An LSTM unit includes a cell state c_t and a hidden state h_t at each time step t. Three multiplicative gates regulate information flow: a forget gate f_t determines what portion of the previous cell state to retain; an input gate i_t and a candidate vector g_t determine what new information to add; and an output gate o_t governs the exposure of the cell state in the hidden state. The standard computation can be described as follows: f_t, i_t, o_t are learned gates that depend on the current input x_t and the previous hidden state h_{t-1}; a candidate vector g_t is generated from x_t and h_{t-1}; the cell state is updated as c_t = f_t ∘ c_{t-1} + i_t ∘ g_t, and the hidden state is h_t = o_t ∘ tanh(c_t). Variants may omit or couple certain gates for efficiency.

Variants and extensions include peephole connections, coupled input and forget gates, and bidirectional or stacked LSTMs. LSTMs are commonly used in combination with attention mechanisms for alignment tasks.

Training is performed via backpropagation through time, often with gradient clipping to mitigate exploding gradients. Regularization such as dropout can be applied carefully. Practical training relies on GPUs and optimized optimizers (e.g., Adam, RMSProp) to handle the computational load of long sequences.

Applications span natural language processing, speech recognition, machine translation, time-series forecasting, video processing, and music generation.