multiplicativerecurrent

Multiplicative recurrent, often referred to as multiplicative recurrent neural networks (MRNNs), is a class of recurrent neural network architectures in which the recurrence relation includes multiplicative interactions between the input and the previous hidden state. This design allows the hidden-to-hidden transition to be modulated by the current input, enabling the model to adapt its dynamics to different contexts within a sequence.

Conceptually, the standard additive recurrence used in many RNNs is replaced by a multiplicative interaction. A common realization uses a factorized or tensor-based weight structure to compute a set of intermediate representations from the previous state and the current input, which are then combined—often through elementwise multiplication—before passing through a nonlinearity to produce the next hidden state. The result is richer, input-dependent dynamics that can capture complex temporal patterns and long-range dependencies more effectively than purely additive recurrences.

History and variants include early explorations of multiplicative interactions in recurrent models, with subsequent work on more efficient factorization and training techniques. Related ideas encompass multiplicative integration approaches that blend input and state contributions in a way that balances expressiveness with parameter efficiency. Training these models can be more challenging due to increased parameter counts and potential training instability, leading to the use of regularization, careful initialization, gradient clipping, and specialized optimization strategies.

Applications and evaluation: Multiplicative recurrent architectures have been studied for sequence modeling tasks such as language modeling, handwriting and music generation, and other sequential domains where transitions depend on context. They are typically compared with vanilla RNNs, LSTMs, and GRUs, and can offer improvements in tasks with strong input-dependent dynamics, at the cost of higher computational requirements and implementation complexity.