informationtheory

Information theory is a branch of applied mathematics and electrical engineering that studies the quantification, storage, and communication of information. It provides a mathematical framework for measuring information and for understanding the limits of data compression and reliable communication. The field emerged in the 1940s, drawing on Ralph Hartley’s idea that information can be measured as a function of the logarithm of the number of possible messages, and was formalized by Claude E. Shannon. Shannon introduced probabilistic models of information sources and channels and established fundamental limits known as coding theorems.

The central concepts include entropy, mutual information, and channel capacity. For a discrete X with distribution p(x), entropy H(X) = - sum_x p(x) log p(x) (base 2, in bits). Mutual information I(X;Y) measures information about X conveyed by Y and is I(X;Y) = H(X) - H(X|Y) = sum_{x,y} p(x,y) log [p(x,y)/(p(x)p(y))]. The channel capacity C is the maximum of I(X;Y) over input distributions. Shannon’s theorems show that a source can be compressed to its entropy rate, and that reliable communication is possible below C but not above it. Rate-distortion theory generalizes to lossy compression, describing trade-offs between rate and distortion.

Applications include data compression (examples such as MP3 and JPEG), error-correcting codes, and the design of communication and storage systems. Information theory is distinct from algorithmic information theory, which studies information content through incompressibility and Kolmogorov complexity.

---