scoringregel

A scoringregel, or scoring rule, is a function used to evaluate the quality of probabilistic forecasts by assigning a numerical value to the forecast together with the observed outcome. Formally, it takes a forecast distribution p over possible outcomes and the actual outcome o, and returns a score S(p, o). In many contexts S is a loss, meaning lower values indicate better forecasts, while in other conventions higher scores are preferred.

A key concept is propriety. A scoring rule is proper if the expected score (or loss) is optimized by reporting the true belief about the outcome distribution. If the optimal value is achieved only when the forecast matches the true distribution, the rule is strictly proper. Propriety encourages honesty and accuracy in probabilistic forecasting, and many scoring rules are designed to be proper or strictly proper.

Common examples include:

- Brier score: a mean squared difference between forecast probabilities and the observed outcome indicators, used for multi-class probability forecasts.

- Logarithmic score (log score): the negative log-likelihood of the observed outcome under the forecast distribution.

- Spherical score: a dot-product-like measure that normalizes the forecast probabilities.

- Continuous ranked probability score (CRPS): a measure suitable for continuous outcomes, comparing the forecast CDF to the observed value.

Applications of scoring rules span meteorology, economics, finance, machine learning, and sports analytics. They are used to compare models, guide calibration and refinement of probabilistic forecasts, and summarize forecast quality over time. Limitations include sensitivity to rare events, dependence on the outcome space, and the need for adequate sample sizes to draw reliable conclusions. See also concepts such as calibration and sharpness in forecast evaluation.