ReLUxp

ReLUxp is a parametric activation function proposed as a generalization of the rectified linear unit (ReLU). It is defined for a real input x and a positive shape parameter p > 0 by the expression f(x) = (max(0, x))^p. This means that for positive inputs, the output is x raised to the power p, while for non-positive inputs the function output remains zero. When p equals 1, ReLUxp reduces to the standard ReLU.

The parameter p controls the curvature of the activation on the positive side and thus influences gradient flow during training. For x > 0, the derivative is f'(x) = p x^{p-1}, and for x < 0, f'(x) = 0. At x = 0, the right-hand derivative is 0 if p > 1, undefined if p < 1, and 1 if p = 1. Consequently, larger values of p dampen gradients near small positive activations, while smaller values of p (0 < p < 1) can amplify gradients near zero and potentially increase instability during training.

ReLUxp is straightforward to implement in modern neural network libraries, requiring a standard max operation followed by a power for positive inputs. Its empirical performance is task-dependent; some studies report modest gains in certain architectures, while others find no consistent advantage over ReLU or other activations. It is not as widely adopted as common variants such as Leaky ReLU or PReLU, which address issues related to negative inputs and gradient flow.

See also: ReLU, Leaky ReLU, PReLU, ELU, Swish, Mish.