Surprisefrom

Surprisefrom is a concept used in probability and information theory to quantify the information revealed by observing a particular outcome relative to a reference probability distribution. The term is not widely standardized, but it appears in discussions that emphasize how the chosen baseline distribution shapes the measure of surprise.

Definition and notation: For a discrete random variable X with outcomes x and a reference distribution P, the surprisefrom of x is defined as S_from(x) = -log_b P(x), where b is the logarithm base (commonly 2 for bits or e for nats). If a contextual or conditional baseline is used, S_from(x|c) = -log_b P(x|c). When P(x) = 0, the surprisefrom is formally infinite. The measure can be extended to distributions and to continuous variables via density-based formulations.

Relation to related concepts: Surprisefrom is fundamentally the same quantity as the standard surprisal or information content of an outcome under P. The expected surprisefrom over P equals the entropy H(P). If one uses a different model Q to evaluate x, the expected value of -log Q(x) under P yields cross-entropy, and the difference between P and Q connects to the Kullback–Leibler divergence.

Applications and variants: Surprisefrom serves as a simple score for model evaluation, anomaly detection, and feature importance in some analytical workflows. It can be context-aware, incorporating conditioning information, and extended to vector-valued or continuous settings through appropriate generalizations such as differential surprisal.

See also: Surprisal, entropy, cross-entropy, Kullback–Leibler divergence, log-likelihood.