Informatsioonihulk

Informatsioonihulk, also known as information content or entropy, is a measure of the uncertainty associated with a random variable. It quantifies the amount of information produced when the value of the random variable is revealed. The concept is fundamental in information theory and has applications in various fields, including data compression, cryptography, and machine learning.

The informatsioonihulk of a discrete random variable X, denoted as H(X), is defined as the expected value of the negative logarithm of the probability mass function of X. Mathematically, it is expressed as:

H(X) = -∑ P(x) log P(x)

where P(x) is the probability of the random variable X taking on the value x, and the summation is over all possible values of X. The base of the logarithm determines the unit of measurement for the informatsioonihulk. Common bases include 2 (bits), e (nats), and 10 (hartleys).

The informatsioonihulk has several important properties. It is always non-negative, with H(X) = 0 if and only if X is a constant (i.e., it takes on only one value with probability 1). It is also additive for independent random variables, meaning that the informatsioonihulk of a pair of independent random variables is the sum of their individual informatsioonihulks.

In the context of data compression, informatsioonihulk provides a lower bound on the average number of bits required to encode the outcomes of a random variable. This is known as the source coding theorem. In cryptography, it is used to quantify the uncertainty of a cryptographic key and the effectiveness of encryption algorithms. In machine learning, informatsioonihulk is used to measure the uncertainty of a model's predictions and to design algorithms for feature selection and dimensionality reduction.