LSTM

Long short-term memory (LSTM) is a type of recurrent neural network designed to learn long-range dependencies in sequence data. LSTMs mitigate vanishing and exploding gradient problems that impede standard recurrent networks during learning, by maintaining a memory cell that can preserve information over many time steps.

An LSTM unit uses a memory cell c_t and a hidden state h_t at each time step t, controlled by three gates: forget gate f_t, input gate i_t, and output gate o_t. The gates decide what information to forget, what to add, and what to expose. A common set of update equations (using sigmoid and tanh activations) is: f_t = sigmoid(W_f x_t + U_f h_{t-1} + b_f); i_t = sigmoid(W_i x_t + U_i h_{t-1} + b_i); c̃_t = tanh(W_c x_t + U_c h_{t-1} + b_c); c_t = f_t ∘ c_{t-1} + i_t ∘ c̃_t; o_t = sigmoid(W_o x_t + U_o h_{t-1} + b_o); h_t = o_t ∘ tanh(c_t).

LSTM networks are typically trained by backpropagation through time (BPTT). They can model long-range dependencies more effectively than standard RNNs, but are more computationally intensive and require careful tuning. Extensions include peephole connections, which let gates access the cell state, and variants such as the Gated Recurrent Unit (GRU), which simplifies the gating structure.

Origins: LSTM was introduced in 1997 by Sepp Hochreiter and Jürgen Schmidhuber. Applications span speech recognition, language modeling, machine translation, time-series forecasting, and video analysis, among others.