KeepProbabilities

Keep probabilities, commonly called keep_prob, refer to the probability that a neuron or unit is kept active during dropout, a regularization technique used to reduce overfitting in neural networks. In training, each unit is independently retained with probability p_keep and zeroed with probability 1 − p_keep, according to a Bernoulli mask. The resulting activations are h' = h ⊙ m, where m is the dropout mask. This stochastic removal of units forces the network to rely on distributed representations rather than any single pathway.

In inverted dropout, the kept activations are scaled by 1/p_keep during training, so that the expected activation remains approximately constant. At test time, dropout is disabled and the full network is used without applying the mask; with inverted dropout no separate scaling is needed during inference.

Typical values for p_keep depend on the layer and dataset. Common choices are around 0.5 for hidden layers and 0.8 to 0.9 for input or convolutional layers. The keep probability is a hyperparameter and should be tuned. Low p_keep strengthens regularization but can impair learning; high p_keep offers less regularization.

Implementation notes vary across frameworks. Many frameworks expose a keep_prob parameter (e.g., TensorFlow’s dropout uses keep_prob), while others use a dropout rate that specifies the probability of zeroing a unit. For recurrent networks, standard dropout can interact with sequence processing, and specialized variants (such as variational dropout) are sometimes preferred to maintain consistency across time steps. Keep probabilities thus form a central, tunable aspect of dropout-based regularization.