CalibrationPlots

CalibrationPlots are graphs used to assess how well probabilistic predictions match observed frequencies. In predictive modeling, calibration refers to the agreement between predicted probabilities and the actual outcomes. A calibration plot for a binary outcome typically shows the relationship between the predicted probability and the observed event rate.

Construction typically involves partitioning predictions into bins by predicted probability (for example, 0–0.1, 0.1–0.2, etc.). For each bin, the mean predicted probability and the observed proportion of positive outcomes are computed. These pairs form points on the plot, which is often accompanied by a reference diagonal line representing perfect calibration (predicted probability equals observed frequency). To obtain a smooth calibration curve, smoothing methods such as LOESS or isotonic regression can be applied across the range of predicted probabilities.

Interpretation focuses on the position of points relative to the diagonal. Points above the diagonal indicate that observed frequencies exceed predictions, meaning the model is underestimating risk in those ranges. Points below indicate overestimation. Overall calibration can be summarized with metrics such as the Brier score, calibration-in-the-large, or calibration slope.

Variants and extensions include reliability diagrams and calibration belts, which provide confidence bands around the calibration curve. For multiclass or probabilistic forecasting, class-wise calibration plots or alternative calibration measures can be used. Calibration plots are commonly employed in machine learning, medicine, meteorology, and finance to diagnose and improve the probabilistic reliability of forecasts, guiding the choice of calibration methods such as isotonic regression or temperature scaling.