SelfNormalizing

Self-normalizing refers to a property of a processing system in which normalized statistical properties are maintained across successive transformations, reducing the need for external normalization steps. In statistics, this can describe data preprocessing or transformations that aim to standardize features to have zero mean and unit variance, improving the stability and performance of learning algorithms.

In deep learning, the term is most closely associated with self-normalizing neural networks (SNNs). SNNs use the Scaled Exponential Linear Unit (SELU) activation function together with a specific weight initialization to promote activations that converge toward zero mean and unit variance as they propagate through layers. The theoretical idea is that, for sufficiently wide networks, the mean and variance of layer activations approach fixed points, preserving stable signal propagation without explicit normalization layers. The SELU parameters are chosen so that the activations tend to stabilize, with typical constants around lambda ≈ 1.0507 and alpha ≈ 1.67326.

Advantages of self-normalizing networks include reduced reliance on batch normalization and improved training stability, particularly for very deep architectures. However, the approach has limitations: it often requires particular architectural choices, sufficient width, and careful initialization; some regularization techniques (such as certain dropout methods) can disrupt the self-normalizing effect; and performance gains are task- and architecture-dependent, not universal.

Self-normalization contrasts with methods like batch normalization, which explicitly normalize activations using batch statistics during training. References to SNNs commonly point to the 2017 work introducing SELU-based self-normalizing networks.