informationcontent

Information content, also known as surprisal, quantifies the amount of information gained when observing a particular outcome. For a discrete random variable X with outcome x, it is defined as I(x) = -log_b P(X = x). The base b determines the units of the measure; base 2 yields bits. More surprising or unlikely outcomes have higher information content, reflecting greater surprise upon observation. The concept was developed within information theory, notably by Claude E. Shannon in the 1940s.

The average information content across all outcomes is the entropy of the source: H(X) = E[I(X)] = - sum_x P(X=x) log_b P(X=x). This quantity captures the expected amount of information produced per symbol. The choice of logarithm base again sets the units (bits for base 2, nats for base e). For a sequence of independent symbols, information contents are additive: I(x1, x2, ..., xn) = sum_i I(xi). For continuous variables, a density-based analogue i(x) = -log f(x) exists, but differential entropy is not a direct counterpart to discrete entropy and carries caveats.

Applications of information content include data compression and coding, where code lengths are related to the information content of outcomes. Rare events, having high information content, influence optimal coding schemes to reduce average message length. In other fields, surprisal is used to model processing difficulty in psycholinguistics and to analyze information structure in text. Related concepts include entropy, cross-entropy, and Kullback–Leibler divergence, which quantify average information and divergences between distributions. The notion of information content thus distinguishes the specific information carried by an observed event from the overall information carried by a source.