The Majority Vote Paradigm Shift: When Popular Meets Optimal

2025-02-18

AI Summary

This study investigates the theoretical optimality of majority voting (MV) for label aggregation in crowdsourcing, aiming to characterize necessary and sufficient conditions under which MV achieves the Bayes-optimal error bound. Method: Integrating statistical learning theory, label noise modeling, and Bayesian decision analysis, we systematically derive optimality criteria for MV under constraints on annotator noise structure and class priors, yielding a verifiable “optimality certificate.” Contribution/Results: We prove that MV is strictly equivalent to the Bayes-optimal estimator when annotator noise satisfies a specific tolerance bound. Empirical validation on synthetic and real-world datasets demonstrates that our criterion accurately predicts performance phase transitions of MV, significantly enhancing label aggregation reliability—without requiring ground-truth labels or additional expert annotations.

Technology Category

• Machine Learning: Ensemble Methods
• Reasoning under Uncertainty: Stochastic Optimization
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Reliably labelling data typically requires annotations from multiple human workers. However, humans are far from being perfect. Hence, it is a common practice to aggregate labels gathered from multiple annotators to make a more confident estimate of the true label. Among many aggregation methods, the simple and well known Majority Vote (MV) selects the class label polling the highest number of votes. However, despite its importance, the optimality of MV's label aggregation has not been extensively studied. We address this gap in our work by characterising the conditions under which MV achieves the theoretically optimal lower bound on label estimation error. Our results capture the tolerable limits on annotation noise under which MV can optimally recover labels for a given class distribution. This certificate of optimality provides a more principled approach to model selection for label aggregation as an alternative to otherwise inefficient practices that sometimes include higher experts, gold labels, etc., that are all marred by the same human uncertainty despite huge time and monetary costs. Experiments on both synthetic and real world data corroborate our theoretical findings.

Problem

• Majority Vote optimality conditions
• Label estimation error bounds
• Annotation noise tolerable limits

Innovation

• Majority Vote optimality
• label estimation error
• annotation noise limits