KLdivergensen

KLdivergensen, commonly known as the Kullback–Leibler divergence, is a measure of how one probability distribution diverges from a second, reference distribution. For discrete distributions over a finite or countable set X, it is defined as D_KL(P||Q) = sum_x P(x) log(P(x)/Q(x)). For continuous distributions with density p and q, it is D_KL(P||Q) = ∫ p(x) log(p(x)/q(x)) dx. The logarithm is usually natural, so the result is in nats; using base 2 yields bits.

Key properties include non-negativity: D_KL(P||Q) ≥ 0, with equality if and only if P and Q are equal almost everywhere (Gibbs’ inequality). The measure is not symmetric, meaning D_KL(P||Q) ≠ D_KL(Q||P) in general. It can be infinite if Q assigns zero probability to events that P assigns positive probability to. In many cases it is convex in the reference distribution Q when P is fixed.

Interpretation and relation to coding: D_KL(P||Q) equals the expected log-likelihood ratio under P. It can be written as the difference between cross-entropy and the entropy of P. Intuitively, it represents the extra expected code length per symbol when using a code optimized for Q to encode samples drawn from P.

Applications: KL divergence is fundamental in information theory, statistics, and machine learning. It underpins likelihood-based methods, hypothesis testing, and model selection. In variational inference and autoencoders, the KL term acts as a regularizer that constrains an approximate distribution to resemble a prior or reference distribution.

History: the measure is named after Solomon Kullback and Richard A. Leibler, who introduced it in 1951. See also Jensen–Shannon divergence and other f-divergences.