relativeentropy

Relative entropy, also known as Kullback-Leibler divergence or information gain, is a measure of how one probability distribution differs from a second, reference probability distribution. It quantifies the "distance" between two probability distributions, although it is not a true metric as it is not symmetric. For two discrete probability distributions P and Q, defined over the same sample space, the relative entropy of Q with respect to P is given by the sum of P(i) times the logarithm of the ratio of P(i) to Q(i) for all possible outcomes i.

In simpler terms, relative entropy tells us how much information is lost when we use distribution Q to approximate distribution P. A higher value indicates a greater difference between the distributions and more information loss. It is widely used in various fields, including machine learning, statistics, and information theory. For instance, in machine learning, it is often used to measure the difference between a model's predicted probability distribution and the true distribution of the data, serving as a cost function for training.

The concept can be extended to continuous probability distributions, where the summation is replaced by an integral. Relative entropy is always non-negative, and it is zero if and only if the two distributions are identical. Its non-symmetric nature means that the relative entropy of Q with respect to P is generally not equal to the relative entropy of P with respect to Q.