WOMAC: A Mechanism For Prediction Competitions

2025-08-25

AI Summary

Standard prediction contests suffer from two fundamental issues: label noise causing weak predictors to occasionally outperform stronger ones, and “winner-takes-all” incentives inducing strategic misreporting. While existing randomized incentive-compatible mechanisms mitigate these problems, they introduce additional noise, compromising determinism and practical usability. This paper proposes WOMAC—the first deterministic, incentive-compatible, and statistically efficient contest mechanism. WOMAC replaces noisy ground-truth labels with a posterior-optimal aggregation of crowd predictions as the scoring benchmark, and employs vectorized computation to ensure scalability and efficiency. Theoretical analysis and empirical evaluation on real-world datasets demonstrate that WOMAC significantly improves expert identification accuracy, result reproducibility, and out-of-sample predictive validity—particularly under high label noise.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Multiagent Systems: Mechanism Design
• Machine Learning: Learning Preferences or Rankings

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Competitions are widely used to identify top performers in judgmental forecasting and machine learning, and the standard competition design ranks competitors based on their cumulative scores against a set of realized outcomes or held-out labels. However, this standard design is neither incentive-compatible nor very statistically efficient. The main culprit is noise in outcomes/labels that experts are scored against; it allows weaker competitors to often win by chance, and the winner-take-all nature incentivizes misreporting that improves win probability even if it decreases expected score. Attempts to achieve incentive-compatibility rely on randomized mechanisms that add even more noise in winner selection, but come at the cost of determinism and practical adoption. To tackle these issues, we introduce a novel deterministic mechanism: WOMAC (Wisdom of the Most Accurate Crowd). Instead of scoring experts against noisy outcomes, as is standard, WOMAC scores experts against the best ex-post aggregate of peer experts' predictions given the noisy outcomes. WOMAC is also more efficient than the standard competition design in typical settings. While the increased complexity of WOMAC makes it challenging to analyze incentives directly, we provide a clear theoretical foundation to justify the mechanism. We also provide an efficient vectorized implementation and demonstrate empirically on real-world forecasting datasets that WOMAC is a more reliable predictor of experts' out-of-sample performance relative to the standard mechanism. WOMAC is useful in any competition where there is substantial noise in the outcomes/labels.

Problem

• Addresses incentive incompatibility in standard competition designs
• Reduces outcome noise impact on expert performance evaluation
• Introduces deterministic mechanism for more reliable winner selection

Innovation

• Scores experts against best peer aggregate predictions
• Uses deterministic mechanism for incentive compatibility
• Provides efficient vectorized implementation for competitions