SqrtReLUx

SqrtReLUx is a nonlinear activation function defined by f(x) = sqrt(max(0, x)). It applies the standard rectified linear unit (ReLU) to x and then takes the square root of the result, yielding zero for nonpositive inputs and a square-root curve for positive inputs.

Definition and basic form

For any real x, f(x) = sqrt(max(0, x)). The domain is all real numbers, and the range is [0, ∞). The function equals 0 when x ≤ 0 and equals sqrt(x) when x > 0.

Key properties

- Monotonicity and continuity: f is nondecreasing and continuous on the real line.

- Differentiability: f is differentiable for x > 0 with f'(x) = 1/(2 sqrt(x)); for x < 0, f'(x) = 0; it is not differentiable at x = 0.

- Shape: the left region is flat at zero (x ≤ 0), and the right region follows a concave sqrt(x) curve for x > 0.

- Second derivative: f''(x) = 0 for x < 0, and f''(x) = -1/(4 x^(3/2)) for x > 0.

Relation to other activations

SqrtReLUx can be viewed as the composition of ReLU with a power-1/2 transform: f(x) = (ReLU(x))^(1/2). It shares nonnegativity with ReLU, but the gradient decreases as x grows, which contrasts with the constant gradient of ReLU for positive inputs.

Usage and variants

In neural networks, sqrtReLU can serve as a pointwise activation with gradually diminishing gradient for large positive x. It is one example in the family f_p(x) = (ReLU(x))^p with 0 < p ≤ 1, where p = 1/2 yields sqrtReLUx. The choice of activation depends on the model and training dynamics.