KreuzentropieLoss

KreuzentropieLoss, commonly referred to in English as cross-entropy loss, is a widely used loss function in supervised learning for probabilistic classification tasks. It measures the distance between a model’s predicted probability distribution over classes and the true distribution, typically represented as a one-hot vector for the target class. The function is derived from maximum likelihood estimation and is closely related to the Kullback–Leibler divergence.

For a single example with true distribution y and predicted distribution p, the loss is L = - sum_i y_i log p_i. If the target is a single class c, and p_i are the predicted probabilities, L = - log p_c. In practice, p is obtained by applying a softmax to the network outputs z: p = softmax(z).

Backpropagation yields simple gradients, with respect to z, proportional to p - y, which facilitates learning.

Variants and implementations: Binary cross-entropy for binary classification; categorical cross-entropy for multi-class single-label; sparse categorical cross-entropy uses integer class labels instead of one-hot vectors. For numerical stability, many frameworks combine softmax and cross-entropy in a single operation. For multi-label problems, binary cross-entropy is typically used per class.

Applications and limitations: It is standard in neural networks for image, text, and general classification tasks. It assumes correctly calibrated probabilities and can be affected by class imbalance; techniques such as class weighting, focal loss, or label smoothing are used to address such issues.