logsumexp

Logsumexp is a mathematical function used to compute the logarithm of a sum of exponentials in a numerically stable way. For a vector x = (x1, ..., xn), it is defined as logsumexp(x) = log(sum_i exp(x_i)). This quantity often appears in log-domain probability calculations, such as log-likelihoods, log-partition functions, and other scenarios involving log-sum-exp terms.

Directly computing exp(x_i) can lead to overflow or underflow, so a common stable form is to subtract the maximum value: m = max_i x_i, then logsumexp(x) = m + log(sum_i exp(x_i - m)). This preserves the correct result while avoiding numerical extremes.

The derivative with respect to x_j is exp(x_j)/sum_i exp(x_i) = softmax_j(x). This creates a close relationship to the softmax function, which is defined as softmax_i(x) = exp(x_i)/sum_j exp(x_j). Logsumexp thus serves as the log of the normalization term in softmax computations.

Extensions and implementations include applying logsumexp along a specified axis for matrices or tensors, yielding a vector of log-sum-exp values. It is implemented in many numerical libraries with vectorized and broadcasting support.

Applications of logsumexp span machine learning and statistics, including computing log-likelihoods in probabilistic models, evaluating partition functions in graphical models, and facilitating numerically stable log-domain computations in neural networks and natural language processing. See also softmax, log-probability, and partition function.