JensenShannon

The Jensen-Shannon divergence, often abbreviated as JSD, is a method for measuring the similarity between two probability distributions. It is a symmetrized, smoothed version of the Kullback–Leibler divergence and is widely used in statistics, information theory, and machine learning. The term reflects its foundations in Jensen’s inequality and Shannon entropy.

For discrete distributions P and Q over the same support, let M be the average distribution M = (P + Q)/2. The Jensen-Shannon divergence is defined as JSD(P || Q) = 1/2 KL(P || M) + 1/2 KL(Q || M), where KL denotes the Kullback–Leibler divergence. With logarithms chosen to base 2, JSD values lie between 0 and 1, and the divergence is finite even if P or Q assigns zero probability to some events. The square root of JSD is a metric, known as the Jensen-Shannon distance.

Key properties include symmetry (JSD(P || Q) = JSD(Q || P)), non-negativity (JSD ≥ 0), and zero if and only if P = Q. The measure is robust to zero probabilities due to the averaging step that produces M. It is finite for all distributions with absolutely continuous components relative to M and is independent of a particular labeling of outcomes.

History and usage: JSD was introduced by Lin in 1991 as a bounded, symmetric measure based on Shannon entropy and Jensen’s inequality. It is widely used to compare probability distributions in applications such as text and image analysis, topic modeling, clustering, and bioinformatics, among others.