AUCul

AUCul, short for Area Under the Calibration Curve, is a metric used to evaluate the calibration quality of probabilistic models. It measures how closely predicted probabilities align with observed outcomes across the probability range, focusing on the accuracy of probability estimates rather than purely on discrimination or ranking.

To compute AUCul, predictions are typically partitioned into bins (for example, deciles). For each bin, the mean predicted probability and the observed event rate are calculated, forming a calibration curve that maps predicted probability to observed frequency. A smoothing step may be applied to reduce noise, after which the area under the resulting calibration curve is computed. In practice, the value is often normalized to facilitate comparison across models or datasets. Uncertainty in AUCul can be assessed via resampling methods such as bootstrapping.

Interpretation and relationship to other metrics: A higher AUCul generally indicates better calibration, meaning the model’s predicted probabilities reflect actual frequencies more accurately. Unlike the Area Under the ROC Curve (AUC), which emphasizes the model’s ability to rank positive cases higher than negative ones, AUCul specifically assesses the reliability of probability estimates. Together with reliability diagrams and calibration plots, AUCul provides a fuller picture of a model’s predictive performance.

Applications: AUCul is used in domains where accurate probability estimates are important, such as healthcare risk prediction, weather forecasting, finance, and any setting where decision thresholds depend on calibrated probabilities. Limitations include sensitivity to the number of prediction instances and to binning choices; smoothing can mitigate sample-size issues but may obscure fine-grained calibration details.

See also: calibration curve, reliability diagram, ROC AUC, probabilistic forecasting.