Labelvary

Labelvary is a term used in data annotation and machine learning to describe the variability observed in label assignments across annotators, contexts, or models. It captures how consistently items are labeled and how the chosen labels are distributed among available categories. While not a formal standard in the literature, labelvary is used to discuss label uncertainty and label noise beyond single-annotator judgments.

Per-item labelvary can be quantified in several ways. One common approach is to compute the entropy of the label distribution across annotators for each item, with higher entropy indicating greater disagreement. Another is the average pairwise disagreement between annotators. At the dataset level, one can report the mean entropy, the overall disagreement rate, or similar statistics such as Gini impurity. Labelvary is related to but distinct from traditional inter-annotator agreement measures; it focuses on the spread of labels rather than a single agreement score.

Labelvary has practical applications in data curation and quality control, helping teams target items for review or adjudication. It informs labeling guidelines and training if certain cases consistently produce diverse labels. It can be used to weight examples during model training, or to select informative items for active learning by prioritizing high-labelvary cases.

Limitations and considerations accompany labelvary. It is sensitive to context, label taxonomy, and annotator instructions, so it should be interpreted alongside guidelines and domain knowledge. There is no universally accepted target level of variation, and different tasks tolerate different amounts of labelvary. It is often used in conjunction with inter-annotator agreement metrics to provide a fuller picture of labeling quality.