sqrtReL

sqrtReL, short for square-root Rectified Linear Unit, is an activation function used in neural networks. It is the composition of the ReLU nonlinearity with the square-root function, defined as f(x) = sqrt(max(0, x)). Equivalently, sqrtReL(x) = sqrt(ReLU(x)).

Mathematical properties

For x < 0, sqrtReL(x) = 0. For x > 0, sqrtReL(x) = sqrt(x). The function is continuous and nondecreasing. It is differentiable for all x ≠ 0, with derivative f'(x) = 0 for x < 0 and f'(x) = 1/(2 sqrt(x)) for x > 0; at x = 0 the derivative is not defined due to the cusp. On the positive side, the derivative grows without bound as x approaches 0, and decreases for larger x. On the positive domain, the function is homogeneous of degree 1/2, since f(tx) = sqrt(t) f(x) for t > 0 and x > 0.

Usage in neural networks

sqrtReL is used as an elementwise activation after linear transformations. Compared with ReLU, sqrtReL introduces a gentler, nonlinear growth for mid-to-large inputs but increases sensitivity near zero due to its steep slope there. This can affect gradient flow: gradients are zero for negative preactivations, while for small positive preactivations they can be large, potentially aiding or destabilizing training depending on the architecture and initialization. Because of its less common usage relative to ReLU, Leaky ReLU, or ELU, empirical results are often mixed and depend on the dataset and network depth.

Relation to other activations

It is related to ReLU via the composition with the square-root function. It preserves nonnegativity like ReLU but differs in smoothness and gradient behavior, offering an alternative nonlinearity for experimentation in network design. See also ReLU and other nonlinear activation functions.