Shannonentrópia

Shannonentrópia, also known as information entropy, is a fundamental concept in information theory and probability theory. It quantifies the uncertainty or randomness associated with a random variable. Introduced by Claude Shannon in his 1948 paper "A Mathematical Theory of Communication," entropy provides a measure of the average amount of information produced by a stochastic source of data.

In simpler terms, entropy can be thought of as the average number of bits required to encode a message from a given source. A source with high entropy is unpredictable, meaning each outcome is equally likely and carries a lot of information. Conversely, a source with low entropy is predictable; some outcomes are much more likely than others, and the information conveyed by each outcome is less.

Mathematically, for a discrete random variable X with possible outcomes x1, x2, ..., xn and corresponding probabilities P(x1), P(x2), ..., P(xn), the Shannon entropy H(X) is defined as the expected value of the self-information of each outcome: H(X) = - Σ P(xi) logb P(xi), where the sum is over all possible outcomes, and b is the base of the logarithm. Commonly, b is 2, resulting in entropy measured in bits, or e, resulting in nats, or 10, resulting in hartleys.

The concept of entropy has wide-ranging applications, including data compression, cryptography, statistical inference, and machine learning. It plays a crucial role in understanding the theoretical limits of communication and data storage.