AUCnon

AUCnon is a generalized evaluation metric in binary classification that extends the conventional area under the ROC curve (AUC) by incorporating abstention or rejection options. It is designed for settings where a model can refrain from making a prediction when confidence is low, and where users must account for the cost of abstention.

AUCnon measures the discriminative ability of a classifier among the subset of instances for which a non-abstaining decision is made, while also incorporating a penalty for abstentions. Intuitively, it evaluates the probability that a randomly chosen positive instance is assigned a higher non-abstention score than a randomly chosen negative instance, conditioned on making a decision. A penalty parameter lambda can be introduced to reflect the cost of abstaining, influencing the trade-off between accuracy and the rate of abstentions.

Computation of AUCnon can be estimated by restricting the standard ROC analysis to the non-abstained predictions, computing the usual AUC on that subset, and optionally adjusting by a penalty term for abstention. Alternatively, it can be formulated as a weighted version of pairwise comparisons where abstentions contribute a reduced weight in the ranking. Estimation often relies on score distributions and, in practice, Monte Carlo sampling or resampling methods.

Relation to AUC: When abstentions are disallowed or the abstention penalty is zero, AUCnon coincides with the traditional AUC. As the abstention cost increases, AUCnon typically decreases, reflecting a preference for fewer but more confident predictions.

Applications and reception: The metric has been discussed in theoretical discussions of decision abstention and in niche machine learning evaluation studies. Critics note that AUCnon introduces a tunable parameter and can be sensitive to its setting, which may complicate cross-study comparisons.