activationfunction

An activation function is a nonlinear function applied to the output of a neuron or to the weighted input plus bias in an artificial neural network. Its primary role is to introduce nonlinearity into the model, enabling it to approximate complex, non-linear relationships rather than just linear mappings. Without activation functions, a multilayer network would collapse into a single linear transform, regardless of the number of layers.

Common activation functions include the sigmoid (logistic) function, tanh, rectified linear unit (ReLU) and its variants, softplus, and softmax. Sigmoid and tanh squash inputs to a finite range and can suffer from vanishing gradients during training. ReLU outputs zero for negative inputs and passes positives through unchanged, offering computational efficiency but sometimes causing dead neurons. Leaky ReLU and other parametric variants aim to mitigate this. Softmax converts a vector into a probability distribution and is typically used in the output layer for multi-class classification. Softplus is a smooth approximation to ReLU.

In practice, activation choices depend on the network architecture and task. Hidden layers commonly use ReLU or its variants, due to efficiency and robust training, while the output layer uses softmax for multi-class classification, sigmoid for binary tasks, or a linear activation for regression. The field has seen numerous alternatives and learnable activations, such as Swish, GELU, and PReLU, reflecting ongoing efforts to improve optimization and generalization.