limitEntropy

LimitEntropy is a term used in information theory and probability to describe the asymptotic behavior of entropy in a sequence of random variables or probability distributions. It is most commonly defined as the entropy rate of a stochastic process: for a process X = {Xn} taking values in a finite alphabet, the limit entropy h is the limit as n grows of (1/n) times the joint entropy H(X1, ..., Xn), provided the limit exists.

Formally, h = lim_{n→∞} (1/n) H(X1, ..., Xn). For stationary processes, this limit exists (by subadditivity and Fekete’s lemma) and equals inf_n (1/n) H(X1, ..., Xn). In the special case of an independent and identically distributed (i.i.d.) process, h equals the single-letter entropy H(X1). The limit entropy per symbol is bounded by 0 ≤ h ≤ log|A|, where |A| is the alphabet size.

LimitEntropy can also refer to the limiting entropy of a sequence of probability distributions p_n on a fixed alphabet, when H(p_n) converges as n grows; in some contexts researchers study the limits of normalized entropies under various transformations or conditioning.

Applications of limitEntropy include characterizing the informational efficiency of data sources, informing compression schemes, and providing a bridge between stochastic processes and coding theory. It is closely related to the concept of entropy rate and is contrasted with single-letter entropy, which measures entropy of a single random variable rather than a long sequence.

See also: Shannon entropy, entropy rate, information rate, ergodic theory, coding theory.