minKL

minKL is a term used in statistics and machine learning to describe optimization procedures that minimize the Kullback-Leibler (KL) divergence between probability distributions. It often arises when one wishes to approximate an intractable target distribution with a more tractable family of distributions, or to fit model parameters to data through a divergence-based objective.

The KL divergence D_KL(P || Q) between two probability distributions P and Q over a common domain quantifies how much P diverges from Q. For discrete distributions, D_KL(P || Q) = sum_x P(x) log(P(x)/Q(x)); for continuous distributions, it is the corresponding integral. The KL divergence is asymmetric: D_KL(P || Q) is generally not equal to D_KL(Q || P). In minKL settings, one typically minimizes D_KL with respect to the parameters of Q (or, in some formulations, with respect to P within a parametric family), depending on which distribution is treated as the fixed target and which is the model to be fitted.

Common applications include variational inference, density estimation, and training certain generative models. In variational inference, a tractable family Q_theta is adjusted to approximate an intractable posterior, often by minimizing D_KL(Q_theta || P) or D_KL(P || Q_theta) as part of an objective such as the evidence lower bound (ELBO). The choice of divergence direction influences the behavior of the approximation, such as which modes are captured or ignored. minKL methods are typically implemented with gradient-based optimization, employing techniques like the reparameterization trick for continuous latent variables.

See also: Kullback-Leibler divergence, variational inference, ELBO, probability density estimation.