InterannotatorAgreement

InterAnnotatorAgreement (IAA) refers to the degree of agreement among independent annotators when labeling data for a task. IAA is a standard measure of reliability and data quality in annotated corpora, surveys, or image datasets, helping determine whether a labeling scheme and guidelines yield consistent results and whether the data are suitable for downstream use. It is distinct from intra-annotator agreement, which assesses an individual’s consistency over time.

Common metrics used to quantify IAA include percent agreement, Cohen’s kappa for two annotators on categorical data, Fleiss’ kappa for multiple annotators, and Krippendorff’s alpha, which generalizes to any number of annotators, data types, and even missing data. Intraclass correlation coefficients (ICC) are often used for continuous or ordinal ratings. Each measure accounts differently for chance agreement and data scale, and the choice depends on the task, the number of annotators, and the annotation level.

Calculation typically involves selecting a representative sample of items, computing the chosen statistics, and reporting uncertainty through confidence intervals. Best practices include clear annotation guidelines, pilot annotation rounds, and an adjudication process to resolve disagreements and produce a gold standard. Reporting should include observed and expected agreement and, when possible, information about the distribution of labels and annotator performance. Awareness of limitations, such as prevalence effects that can bias kappa values, is important for accurate interpretation.

IAA informs dataset quality, guides annotation protocol design, and influences model training and evaluation across fields such as natural language processing, biomedical annotation, and image labeling.