ELUs

ELUs, short for Exponential Linear Units, are a family of activation functions used in artificial neural networks. An ELU with parameter alpha > 0 is defined as f(x) = x for x > 0 and f(x) = alpha*(exp(x) - 1) for x <= 0. The function is continuous and differentiable for all x. Common practice uses alpha = 1.0, though alpha can be treated as a fixed hyperparameter or learned during training.

Compared with the rectified linear unit (ReLU), ELUs produce negative outputs, which helps reduce bias shifts in activations and can lead to faster convergence. The smooth saturation for negative inputs and linear behavior for positive inputs combine to maintain strong gradients while keeping activations centered near zero. ELUs have been applied in various neural network architectures, including feedforward networks, convolutional networks, and recurrent networks, with reported improvements on several benchmark tasks when carefully tuned.

Limitations include higher computational cost due to the exponential term and potential sensitivity to the alpha parameter. In some tasks ELU offers little or no advantage over simpler activations like ReLU, so practitioners choose based on empirical results and resource constraints. In specific self-normalizing network designs, alternative activations such as SELU may be preferred.

Beyond neural networks, the acronym ELU has other meanings in different fields. This article focuses on the activation function; if another sense is intended, please refer to context or a disambiguation page.