zoneout
Zoneout is a regularization technique for recurrent neural networks that aims to improve generalization by introducing stochastic temporal continuity in the hidden states. Unlike standard dropout, which randomly zeros activations, zoneout randomly preserves a subset of the previous hidden states (and, for LSTMs, the cell states) at each time step.
Mechanism-wise, during training a binary mask selects which units should keep their previous values. The updated
Inference commonly uses a deterministic recurrence. At test time, the stochastic masks are typically not sampled;
Hyperparameters and practice vary, but zoneout is typically controlled by a small probability p that governs