Shannonentropie

ShannonEntropie, in information theory often called Shannon entropy, is a measure of the average uncertainty or information content of a discrete random variable X with probability mass function p(x). It is defined by H(X) = - sum over x of p(x) log_b p(x). The choice of the logarithm base b determines the units: base 2 yields bits, base e yields nats. In practice, the base 2 form is common when discussing data compression and coding.

Several key properties characterize Shannon entropy. It is nonnegative and equals zero if and only if X is a certain (degenerate) constant variable. For a finite alphabet, the maximum entropy is achieved for a uniform distribution, with H(X) = log_b |X|. Entropy is concave in the distribution p(x) and satisfies the chain rule: H(X,Y) = H(X) + H(Y|X). It also leads to the concept of mutual information I(X;Y) = H(X) - H(X|Y), which measures the reduction of uncertainty about X given Y.

Interpretation and implications. Shannon entropy represents the average amount of information produced by observing the outcome of X. In coding theory, it sets a fundamental limit: the average length of an optimal lossless code cannot be smaller than H(X) bits per symbol (source coding theorem). For continuous variables, a related concept is differential entropy h(X) = - ∫ f(x) log f(x) dx, which behaves differently and can be negative or not invariant under coordinate changes.

Applications span data compression, communications, cryptography, and statistical modeling, where entropy quantifies unpredictability, diversity, or information content of sources.