Logsoftmax

Logsoftmax is the logarithm of the softmax function. Given a real-valued vector x with length n, the softmax components are softmax_i(x) = exp(x_i) / sum_j exp(x_j). The logsoftmax components are logsoftmax_i(x) = log(softmax_i(x)) = x_i − log(sum_j exp(x_j)). Equivalently, logsoftmax_i(x) = x_i − logsumexp(x), where logsumexp(x) = log(sum_j exp(x_j)).

Numerical stability is a key motivation for using logsoftmax. Directly computing exp(x_i) can overflow for large x_i. The logsumexp trick mitigates this by computing logsumexp(x) in a stable way, for example by subtracting the maximum x_i before exponentiation: logsumexp(x) = max_i x_i + log(sum_j exp(x_j − max_i x_i)).

Derivatives of logsoftmax have a convenient form. For i and j in {1,...,n}, the partial derivative is ∂ logsoftmax_i(x) / ∂ x_j = δ_{ij} − softmax_j(x), where δ_{ij} is the Kronecker delta and softmax_j(x) is exp(x_j)/sum_k exp(x_k). This leads to a simple Jacobian structure often exploited in optimization.

Applications and implementation details: logsoftmax is typically used to produce log-probabilities for each class in classification models. When paired with a negative log-likelihood loss, it avoids computing the softmax probabilities explicitly and improves numerical stability. Many deep learning frameworks provide optimized log_softmax functions that combine the log and softmax computations efficiently.