Selu

SELU, short for Scaled Exponential Linear Unit, is an activation function designed to promote self-normalizing behavior in deep neural networks. It was introduced in 2017 by Klambauer, Unterthiner, Mayr, and Nadler in the paper Self-Normalizing Neural Networks. The goal is to keep activations with near-zero mean and unit variance across layers, reducing or eliminating the need for explicit normalization layers in some architectures.

Mathematically, SELU is defined as follows: f(x) = lambda * x for x > 0, and f(x) = lambda * alpha * (exp(x) - 1) for x ≤ 0. The standard constants are lambda ≈ 1.0507 and alpha ≈ 1.67326. The function is continuous and differentiable at zero. With zero-mean, unit-variance inputs and weights initialized from a LeCun normal distribution, SELU tends to preserve normalized statistics through the network.

SELU is intended to enable self-normalizing neural networks, which can reduce reliance on batch normalization in certain deep architectures. It is commonly used in deep feedforward and convolutional networks. A related technique, AlphaDropout, preserves the self-normalizing property by keeping the mean and variance of activations stable when dropout is applied.

Practical considerations include ensuring inputs are standardized and using the recommended initialization. While SELU can improve convergence and gradient flow in suitable models, its benefits depend on consistent network design and the absence of factors that disrupt the self-normalizing dynamics.