querybycommittee

Query by committee (QBC) is an active learning strategy that uses disagreement among a group of models, or a committee, to select the most informative unlabeled examples for labeling. The central idea is that instances on which the committee members disagree are likely to be informative for improving the predictor, thereby reducing labeling effort.

Typically, QBC starts with a small labeled dataset and a larger pool of unlabeled data. A diverse committee of models is trained on the labeled data. For each unlabeled example, every committee member makes a prediction (hard labels or probability distributions). A disagreement score is computed for each example, using measures such as vote entropy (the entropy of the label votes across the committee) or other metrics like average pairwise disagreement or divergence between members. The examples with the highest disagreement are selected for labeling and added to the labeled set. The process iterates as more labeled data become available, until the labeling budget is exhausted or performance converges.

Key aspects include the composition of the committee (the models can differ in algorithms, training subsets, or hyperparameters) and the choice of disagreement metric. QBC has been applied across domains such as text classification, image recognition, and medical diagnosis. Variants include using bagging or boosting to create diverse committee members and employing soft predictions rather than hard labels.

QBC was introduced in the early 1990s by Seung, Opper, and Sompolinsky, and has since become a foundational approach in active learning for selecting informative samples and reducing labeling costs, albeit at the expense of increased computation due to training multiple models.