Krydsentropi

Krydsentropi, also known as cross-entropy, is a measure used in information theory and machine learning to quantify the difference between two probability distributions. It is a fundamental concept in the fields of statistics, data compression, and predictive modeling.

In information theory, cross-entropy is used to measure the inefficiency of predicting a random variable using a given probability distribution. It is particularly useful in the context of data compression, where it helps in understanding the amount of information lost when using an approximate model to represent the true distribution of data.

In machine learning, cross-entropy is commonly used as a loss function in classification tasks. It measures the performance of a classification model whose output is a probability between 0 and 1. The cross-entropy loss increases as the predicted probability diverges from the actual label. For instance, if the actual label is 1 and the predicted probability is 0.1, the loss will be high, indicating a poor prediction. Conversely, if the predicted probability is close to the actual label, the loss will be low, indicating a good prediction.

The formula for cross-entropy loss between a true distribution \( p \) and an estimated distribution \( q \) is given by:

\[ H(p, q) = -\sum_{i} p(i) \log q(i) \]

where \( p(i) \) is the true probability of class \( i \) and \( q(i) \) is the predicted probability of class \( i \).

Cross-entropy is a versatile and widely used metric due to its ability to provide a clear and interpretable measure of model performance, making it an essential tool in both theoretical and applied contexts.