Distributiondistance

Distributiondistance is a general concept in probability and statistics referring to a numerical quantity that measures how different two probability distributions are, defined on a common measurable space. It is not a single universal measure but a family of metrics and divergences used to compare distributions in theory and practice.

Common measures include the total variation distance d_TV(P,Q) = sup_A |P(A) - Q(A)|; the L1 distance between density functions ∫ |p(x) - q(x)| dx when densities exist; the Kullback–Leibler divergence D_KL(P || Q), which is not symmetric and does not satisfy the triangle inequality; the Jensen–Shannon divergence D_JS(P || Q), which is symmetric and bounded, with its square root often used as a metric; the Wasserstein (Earth Mover’s) distance W_p(P,Q) for p ≥ 1; and the Hellinger distance H(P,Q) = (1/√2) ||√p - √q||_2. Distances typically require a common support or appropriate extension to compare distributions with different supports.

Estimation and computation often rely on empirical or parametric representations. Distances between empirical distributions can be computed directly in discrete cases, or estimated via density estimation, resampling, or numerical algorithms for optimal transport in the Wasserstein case. The choice of distance influences sensitivity to tails, support, and scale.

Applications include hypothesis testing, model selection, detecting distributional shift in domain adaptation, evaluating generative models, and anomaly detection. In practice, the selection hinges on properties such as symmetry, bounds, and computational feasibility.

See also: probability distribution, divergence measures, metric, total variation distance, Wasserstein distance, KL divergence, Jensen–Shannon divergence.