RTRL

Real-Time Recurrent Learning (RTRL) is an online gradient-based learning algorithm for training recurrent neural networks. It computes the gradient of the instantaneous loss with respect to all network weights in real time, allowing weight updates at each time step without backpropagation through time (BPTT).

How it works in brief: For a recurrent network with hidden state h_t that depends on the previous state h_{t-1}, the current input x_t, and the weight matrices, RTRL maintains sensitivity variables that track how h_t changes with each weight. By applying the chain rule through the network’s recurrence, these sensitivities are updated at every time step, and the gradient of the current loss with respect to the weights is formed from the sensitivities and the current error. The weight update is then performed by a standard online rule, such as W ← W − η ∂L_t/∂W, where η is the learning rate.

Complexity and limitations: RTRL requires maintaining and updating a large set of derivatives, which for fully connected recurrent layers leads to substantial computational and memory costs. The complexity of updating the recurrent weight derivatives grows rapidly with the number of hidden units, making RTRL impractical for large networks. This high cost is a major reason why RTRL is less common in modern deep learning architectures.

Relation to other methods: RTRL is the online counterpart to BPTT, which unrolls the network through time and computes gradients via backpropagation across multiple time steps. While BPTT can be more efficient for long sequences, RTRL provides exact online gradients without time-unfolding. Variants and approximations, such as Unbiased Online Recurrent Optimization (UORO), have been developed to reduce the computational burden of online recurrent learning.

History and usage: RTRL was introduced in the late 1980s by Williams and Zipser and has contributed to theoretical understanding of online learning in recurrent networks. In practice, it is largely superseded by BPTT and contemporary online approximation methods for larger networks.