ReLUx12

ReLUx12 is a hypothetical activation function proposed as a higher-order extension of the Rectified Linear Unit (ReLU). It applies a polynomial power to the positive part of the input, defined as f(x) = (max(0, x))^12. For non-positive inputs, the output is zero; for positive inputs, the output is x raised to the 12th power. This makes the function monotone and strictly increasing on the positive side, with strong curvature.

The derivative of ReLUx12 is f′(x) = 12x^11 for x > 0 and f′(x) = 0 for x ≤ 0. Consequently, the gradient is extremely small near zero (due to the x^11 term) and grows rapidly for large positive inputs. This creates a trade-off: the activation can model highly non-linear relationships, but training can be sensitive to initialization and normalization, with a risk of unstable updates if inputs frequently reach large positive values.

In practice, ReLUx12 is mainly discussed in theoretical or toy-setting contexts to study how high-order nonlinearities affect learning dynamics, representation capacity, and gradient flow. It is not broadly adopted in mainstream architectures, where concerns about vanishing gradients near zero and potential gradient explosion at scale favor more moderate activations or learnable slopes.

Variants and generalizations include ReLUxk for other exponents k ≥ 1, with k = 1 recovering the standard ReLU. When using ReLUx12, practitioners typically pair it with normalization schemes, careful weight initialization, or architectural techniques (such as residual connections) to mitigate training instability.

See also: ReLU, Leaky ReLU, PReLU, ELU, Softplus.