ReLUxk

ReLUxk is a family of activation functions used in artificial neural networks, extending the standard rectified linear unit (ReLU) by introducing a nonlinear exponent parameter k > 0. The common mathematical form is f_k(x) = (max(0, x))^k. When k = 1, this reduces to ReLU. For x > 0 it yields x^k, and for x ≤ 0 it yields 0.

Behavior and gradients: For k > 1, small positive inputs have near-zero gradients since the derivative is k x^{k-1}. Larger positive inputs drive steeper growth. For 0 < k < 1, gradients are larger near zero and the positive branch grows sublinearly with x. The derivative is f'_k(x) = k x^{k-1} for x > 0; at x = 0 the subderivative depends on k: it is 0 when k > 1, tends to be undefined for 0 < k < 1, and is typically 1 in the standard subgradient sense at k = 1.

Implementation and stability: The function is implemented with a standard max(0, x) operation followed by a power, which is straightforward in modern frameworks. Numerical stability considerations include handling x = 0 when k < 1 and ensuring stable backpropagation through the exponent, particularly for very small or large inputs.

Usage and context: ReLUxk is discussed as a lightweight parametric alternative to ReLU that allows tuning of gradient flow and activation characteristics for specific tasks or architectures. It preserves non-negativity and can be used similarly to ReLU in deep networks, but may require careful hyperparameter tuning and initialization to manage potential vanishing or exploding gradients depending on k and the data.

See also: ReLU, Leaky ReLU, PReLU, Softmax, activation functions.

---