Softsign

Softsign is a smooth activation function used in neural networks. It is defined by f(x) = x / (1 + |x|). The function maps the entire real line to the interval (-1, 1) and is monotone increasing, with f(0) = 0. The derivative is f'(x) = 1 / (1 + |x|)^2, which is positive for all x and tends to zero as |x| grows, ensuring a saturating behavior for large inputs but with a relatively gentle gradient decay compared to some other saturating activations.

Properties of softsign include its smoothness and differentiability at all points, including zero. Because it is bounded and centered at zero, it can help maintain stable activations without the sharp transitions seen in piecewise linear functions. Compared with tanh, softsign saturates more gradually for large |x| due to its polynomial decay of the gradient (approximately 1/x^2 rather than an exponential decline). It is also differentiable everywhere, unlike some activations that rely on non-differentiable points.

In practice, softsign has been explored as an alternative to tanh or sigmoid activations in both feedforward and recurrent networks. It is simple to implement and incurs similar computational cost to other elementwise activations. However, in modern deep learning, it is less commonly used than ReLU and its variants, as those often offer better training dynamics for deep architectures. Still, softsign remains a useful option in certain experiments and smaller networks.