RNN

A recurrent neural network (RNN) is a class of artificial neural networks designed to handle sequential data. Unlike feedforward networks, RNNs maintain a hidden state that can carry information across time steps, allowing the network to incorporate context from previous inputs. At each time step t, the network receives an input x_t and updates its hidden state h_t based on h_{t-1} and x_t. This architecture makes RNNs suitable for tasks where the order and history of inputs matter.

In the simplest form, the hidden state is updated by h_t = f(W_xh x_t + W_hh h_{t-1} + b_h), and an output y_t can be produced by y_t = g(W_hy h_t + b_y). The same weights are shared across time steps, enabling the model to handle sequences of varying length and to learn temporal dependencies.

Training is performed by backpropagation through time, which unfolds the network across the sequence and applies gradient descent to update parameters. RNNs can suffer from vanishing or exploding gradients when learning long-range dependencies, an issue that motivates the use of specialized architectures such as long short-term memory networks (LSTMs) and gated recurrent units (GRUs).

Common variants include bidirectional RNNs, which process input in forward and backward directions, and encoder–decoder or sequence-to-sequence models for tasks like machine translation. Attention mechanisms and hybrid architectures have further extended their capabilities. RNNs have been applied to language modeling, speech recognition, time series forecasting, and video analysis.

Despite their strengths, RNNs can be computationally intensive and require careful design to capture long-range structure. In many modern applications, transformer-based architectures have supplanted RNNs for sequence modeling due to parallelizability and strong performance.