Logsumexpx

Logsumexpx, usually written logsumexp(x), denotes the natural logarithm of the sum of exponentials of the components of x. Formally, for a real-valued vector x with elements x_i, LSE(x) = log(sum_i exp(x_i)). If x is a scalar, LSE(x) reduces to log(exp(x)) = x. The function can be applied along a specified axis for vectors, matrices, or higher-dimensional arrays, producing a reduced array.

Numerical stability is a key consideration. Direct computation of sum_i exp(x_i) can overflow for large x_i. A common stable form uses the maximum m = max_i x_i: LSE(x) = m + log(sum_i exp(x_i − m)). This prevents overflow and is widely used in practice. The same trick enables stable computations when combining log-sum-exp with other log-domain quantities.

Properties include invariance under additive shifts: LSE(x + c) = LSE(x) + c. The gradient with respect to x_j is exp(x_j) / sum_i exp(x_i), which is the j-th component of the softmax function. Thus LSE is convex in x and smoothly approximates the maximum of the x_i.

Extensions and implementations vary by framework. In matrices, LSE can be computed along a chosen axis, reducing the dimensionality accordingly. Related to softmax, log-domain probabilities, and normalization terms in log-likelihood calculations, LSE is used in statistics, machine learning, and probabilistic modeling. Practical libraries provide efficient implementations, such as scipy.special.logsumexp in SciPy, numpy equivalents, and logsumexp utilities in PyTorch and JAX.