Lowerentropy

Lowerentropy is a term used in information theory and related fields to describe a relative measure of decreased uncertainty in a system or distribution compared with a reference state. It is not a standard, universally defined quantity, but a convenient way to discuss how much entropy has been reduced relative to a baseline.

Definition and interpretation

For a discrete probability distribution P over a finite alphabet, the Shannon entropy is H(P) = -sum p_i log p_i. Given a reference distribution P0 with entropy H0 = H(P0), the lowerentropy of P with respect to P0 can be defined as L(P|P0) = max(0, H0 - H(P)). If H(P) ≤ H0, L is positive and indicates a reduction in uncertainty relative to the baseline; if H(P) ≥ H0, L is zero. A variant without the max, L = H0 - H(P), is also used when the domain is restricted to states where H ≤ H0.

Related concepts

Lowerentropy is related to information gain and entropy reduction concepts used in learning, data compression, and thermodynamics. It can be extended to continuous distributions and to alternative entropy measures such as Rényi entropy. Like other relative measures, its value depends on the chosen reference distribution.

Examples

A degenerate distribution (all probability on one outcome) has H = 0, which yields a large lowerentropy relative to a nontrivial baseline. A uniform distribution on n outcomes has maximal entropy Hmax = log n, so lowerentropy relative to a baseline with lower entropy reflects how much order has been introduced beyond that baseline.

Limitations

Because lowerentropy depends on the reference state, different baselines can yield different values for the same distribution. It is a descriptive, not a universal, metric. See also Shannon entropy, cross-entropy, and KL divergence.