qrels

Qrels, short for query relevance judgments, are a collection of human judgments that indicate the relevance of documents to a set of search queries. They serve as ground truth in the evaluation of information retrieval systems, enabling researchers to quantify how well a system's ranked results match user expectations.

Each line in a qrels file usually records a topic or query identifier, an iteration number (often 0), a document identifier, and a relevance grade. The relevance grade is typically an integer on a predefined scale, such as 0 for not relevant, 1 for relevant, with higher numbers representing more relevance. In some datasets, the scale is binary (0/1) or extended to 3 or more levels.

Qrels are generated through human annotation, often by trained assessors or crowdsourcing, sometimes using pooling methods to limit the number of documents examined. Quality control measures include inter-annotator agreement checks and adjudication. Judgments may be incomplete; not all relevant documents may be graded for every query.

Evaluation uses qrels to compute metrics such as precision at k, recall, average precision, mean average precision (MAP), and normalized discounted cumulative gain (NDCG). The retrieved list is compared against the qrels to produce scores for each query, which are then averaged across queries.

Qrels are dataset-specific; examples include TREC collections, LETOR, and MSMARCO. While the general concept is common, formats and scoring schemes vary between datasets. Limitations include judgment subjectivity and pooling bias, and evolving relevance definitions as information needs change.