sparsemax

Sparsemax is a nonlinear activation and a sparse counterpart to softmax, introduced by André F. Martins and Francisco J. Astudillo in 2016. It is designed to produce probability-like outputs that are often sparse, meaning many components can be exactly zero. This makes sparsemax useful in neural networks and attention mechanisms where it is desirable to focus on a subset of elements.

Mathematically, for an input vector z in R^K, sparsemax(z) is defined as the Euclidean projection of z onto the probability simplex:

sparsemax(z) = argmin_p ||p − z||^2 subject to p ≥ 0 and sum(p) = 1.

Equivalently, the result is a vector p with p_i = max(z_i − τ, 0) for all i, where τ is a threshold chosen so that the components sum to 1. The standard way to compute τ is to sort z in decreasing order, find the largest k such that z_(k) > (sum_{i=1}^k z_(i) − 1)/k, set τ = (sum_{i=1}^k z_(i) − 1)/k, and then apply p_i = max(z_i − τ, 0). The output is a sparse probability distribution, with potentially several zero entries.

Properties and usage: compared with softmax, sparsemax yields sparse outputs, which can enforce selectivity in attention and other predictive tasks. It is differentiable almost everywhere, enabling backpropagation, and its Jacobian has a simple form within the active set where p_i > 0. Sparsemax is part of a broader family of functions (entmax) that interpolates between softmax and sparsemax, allowing controlled sparsity in the output distribution. It is commonly used in structured prediction and attention models where sparse, interpretable attention weights are advantageous.