PReLU

PReLU, or parametric rectified linear unit, is an activation function used in neural networks. It generalizes the standard rectified linear unit by allowing a small, learnable slope for negative inputs. The function is defined as f(x) = x if x is nonnegative, and f(x) = a x if x is negative, where a is a parameter learned during training. The derivative with respect to x is 1 for nonnegative inputs and a for negative inputs, while the derivative with respect to a is x for negative inputs and 0 otherwise.

Variants of PReLU differ in how the parameter a is shared. A common approach uses a separate a for each channel (channel-wise), a single a for the entire layer (layer-wise), or a single a shared across all layers. A widely used practice is to initialize a at a small positive value (often around 0.25) and learn it jointly with other weights. Leaky ReLU is a related fixed-slope variant, where a is constant rather than learnable.

PReLU was introduced to address limitations of ReLU, notably the dying ReLU problem and potential slow convergence in very deep networks. It can improve performance on large datasets and deeper architectures by allowing a small, nonzero gradient for negative activations. However, it adds extra parameters, increasing model capacity and the risk of overfitting on smaller datasets. Training requires proper regularization and monitoring of the learned a values.

In practice, PReLU is supported in many deep learning frameworks and can be applied to convolutional or fully connected layers. Its usefulness is problem-dependent, and it is one of several learnable activation options used to enhance model expressiveness.