SubsetAccuracy

Subset accuracy, also known as the exact match ratio, is a metric used in multi-label classification to evaluate how often the predicted set of labels for a given instance exactly matches the true set of labels.

Definition: For a dataset with N instances, each instance i has a true label set Yi and a predicted label set Ŷi. Subset accuracy is defined as (1/N) times the number of instances for which Ŷi equals Yi, i.e., SubsetAccuracy = (1/N) * sum over i of I(Ŷi = Yi), where I is the indicator function. In practice, label sets can be represented as binary vectors of length L (the total number of possible labels); in this form, accuracy increases only when the prediction vector exactly matches the ground truth vector.

Interpretation and properties: Subset accuracy is a very strict metric. It rewards perfect predictions per instance and is sensitive to errors in any included or missing label. Its value lies between 0 and 1, where 1 indicates all instances are predicted exactly. It is commonly used in scenarios where exact label combinations are crucial.

Relation to other metrics: Unlike per-label metrics such as accuracy, precision, recall, and F1, subset accuracy requires the entire label set to be correct for an instance. It is related to, but more stringent than, set-based measures like Jaccard similarity or Hamming loss, which allow partial credit for partial correct predictions.

Considerations: Subset accuracy can be harsh in datasets with many possible labels or imbalanced label distributions. It is often complemented by other metrics to provide a fuller evaluation of multi-label performance. Thresholds and post-processing steps for converting predicted scores to label sets can also influence the reported subset accuracy.