accuracywhether

Accuracywhether is a proposed evaluation metric used to quantify the performance of systems that provide yes/no answers or yes/no components within broader queries. It focuses on how often a model’s predicted binary response matches the ground truth, providing a straightforward measure of binary decision accuracy.

Computation and variants: For a test set of N instances, each with a true label t_i in {yes, no} and a predicted label p_i, accuracywhether = (1/N) times the number of correct predictions, i.e., the fraction of i for which p_i equals t_i. Some implementations handle abstentions or indeterminate answers differently: abstentions may be counted as incorrect, excluded from the calculation, or reported as a separate metric. When abstentions are excluded, the score reflects only instances where the model produced a definitive yes/no answer.

Origins and usage: The term accuracywhether has appeared in informal discussions and some methodological write-ups as a way to distinguish yes/no evaluation from broader accuracy metrics. It is not a single, universally standardized benchmark, and its exact definition can vary across studies or applications.

Applications: Accuracywhether is applicable to evaluating chatbots, information retrieval systems that return binary decisions, medical decision-support tools, and any natural language processing task that requires a reliable yes/no determination.

Limitations: As a binary-only metric, it does not capture calibration, confidence, or the distribution of errors across classes beyond overall accuracy. It also depends on the clarity of the yes/no labels and can be affected by dataset bias or ambiguity in ground-truth labeling.

Example: A model answers 85 of 100 yes/no questions correctly, yielding an accuracywhether of 0.85.