LeakyReLU

LeakyReLU is a nonlinear activation function used in artificial neural networks, introduced as a variant of the rectified linear unit (ReLU). It is defined as f(x) = x if x > 0, and f(x) = alpha * x if x <= 0, where alpha is a small positive constant, commonly around 0.01.

The function aims to address the dying ReLU problem, where neurons can become inactive if many inputs are negative and therefore never update. By allowing a small, nonzero gradient when the input is negative, LeakyReLU helps maintain gradient flow during training. Its derivative is f'(x) = 1 for x > 0 and f'(x) = alpha for x < 0; at x = 0 the derivative is not defined, but a subgradient can be used in practice.

Typical values for alpha range from 0.01 to 0.3, with 0.01 being a common default. LeakyReLU is simple to implement and computationally inexpensive, adding no significant overhead to standard ReLU networks. It can be used in various architectures, including feedforward networks, convolutional neural networks, and recurrent networks.

Variants and related concepts include Parametric ReLU (PReLU), where alpha is a learnable parameter, allowing the model to adapt the negative slope during training; and Randomized Leaky ReLU (RReLU), which uses a random alpha during training and a fixed, expected value during testing. LeakyReLU is often favored for faster convergence in some tasks but does not always outperform ReLU; in some scenarios, the benefits depend on the data and architecture.

See also: ReLU, PReLU, RReLU, activation functions.