informationdensity

Information density is a measure used in information theory and related disciplines to quantify how much information a unit of data carries. It can be considered per symbol, per character, per time interval, or per unit of bandwidth or space.

In formal terms, the self-information of a specific outcome x with probability p(x) is I(x) = -log2 p(x) bits. The average information per symbol for a source X is its entropy H(X) = -Σ p(x) log2 p(x). Information density can be viewed as the information content relative to the unit of measurement; for a given source the average density is bounded by the entropy. Higher information density means more information per unit, but often comes with higher demands on bandwidth and error protection. Redundancy can improve reliability, while coding schemes seek to compress data toward the source’s entropy while maintaining fidelity.

In natural language, information density captures how much information each word or syllable conveys. It is influenced by predictability: highly expected words carry less surprisal. Density varies with genre, style, context, and discourse structure, affecting readability and processing effort.

Applications include data compression, channel coding, text analytics, and natural language processing. The concept also underpins rate-distortion tradeoffs in multimedia, where one balances information density with perceptual quality. See also entropy, information content, surprisal, mutual information, and coding theory.