selfinformation

Self-information, also called surprisal, is a measure of the information content associated with a particular outcome of a random variable. It quantifies how unexpected or informative an event is given its probability distribution.

For a discrete random variable X with outcomes x and probability p(x), the self-information of x is I(x) = -log_b p(x). The common choice is base 2, which yields units of bits; natural logarithms give units called nats.

The average self-information across all outcomes is the entropy H(X) = E[I(X)] = - sum_x p(x) log_b p(x). The entropy represents the expected amount of information produced per outcome and depends on the spread of the distribution. It is minimized by a degenerate distribution (always the same outcome) and maximized by a uniform distribution over the support.

Key properties include that I(x) ≥ 0, with equality only if p(x) = 1; I(x) increases as the probability of the outcome decreases. For independent events, self-information is additive: I(x1, x2) = I(x1) + I(x2). In coding terms, H(X) sets a lower bound on the average length of any lossless code for X.

Extensions include continuous variables, where a related quantity i(x) = -log f(x) uses the probability density f(x). Unlike discrete self-information, differential quantities require careful treatment and do not form a true probability measure by themselves, but the underlying idea remains the same. Self-information underpins core results in information theory, including data compression and coding efficiency.