zoneout

Zoneout is a regularization technique for recurrent neural networks that aims to improve generalization by introducing stochastic temporal continuity in the hidden states. Unlike standard dropout, which randomly zeros activations, zoneout randomly preserves a subset of the previous hidden states (and, for LSTMs, the cell states) at each time step.

Mechanism-wise, during training a binary mask selects which units should keep their previous values. The updated state is thus a mixture of the newly computed state and the carried-over previous state. This creates noise that preserves information across time and encourages the network to rely on stable temporal patterns rather than rapidly fluctuating activations. Zoneout can be applied to the hidden state in recurrent layers, and in LSTMs it can be used separately on the hidden state h_t and the cell state c_t; it is compatible with other recurrent units such as GRUs as well.

Inference commonly uses a deterministic recurrence. At test time, the stochastic masks are typically not sampled; instead the network uses either the expected values or simply proceeds without applying zoneout, resulting in a stable, non-random computation.

Hyperparameters and practice vary, but zoneout is typically controlled by a small probability p that governs how many units preserve their previous state. Common applications include natural language processing, speech, and time-series tasks, particularly when training data is limited and overfitting is a concern. Benefits include improved generalization and temporal robustness; limitations involve the need to tune p and potential changes to training dynamics, which may require careful integration with other regularization strategies.