Finding the Signal in the Spam: Jointly Learning Rewards and Worker Reliability from Pairwise Comparisons

2026-08-10

AI Summary

This work addresses the challenge of data noise in crowdsourced pairwise comparisons arising from unreliable workers—such as those lacking domain expertise or engaging in malicious behavior—by proposing a novel joint learning framework that simultaneously estimates item utilities and worker reliability. The approach extends the Bradley–Terry–Luce model by incorporating worker ability parameters, thereby integrating worker reliability modeling into pairwise comparison learning for the first time. Leveraging Polya–Gamma data augmentation, the method transforms the logistic likelihood into a conditionally Gaussian form, which simplifies the E-step of the EM algorithm and recasts the problem as a matrix sensing task, offering both computational efficiency and theoretical convergence guarantees. Experiments on real-world and synthetic datasets demonstrate that the proposed method significantly outperforms existing baselines and exhibits strong robustness against both spamming and adversarial workers.

Abstract

The problem of learning from pairwise comparisons has been widely studied across many domains such as recommendation systems, social choice, and more recently, fine-tuning large language models. In this problem, the goal is to learn item rewards based on pairwise comparisons between them. In many scenarios, these comparisons are elicited from crowdworkers using platforms such as Amazon Mechanical Turk, Scale AI, etc. However, crowdworkers are often unreliable due to limited domain knowledge or revenue-maximizing (spamming) behavior. In this work, our goal is to understand whether worker reliability (competency) can be learned jointly with item rewards. To this end, we adopt the Boltzmann-rational model for pairwise comparisons, which extends the Bradley-Terry-Luce model by incorporating worker competencies. We derive an EM-based algorithm for learning under this model by introducing Polya-Gamma latent variables to transform the logistic likelihood into a conditionally Gaussian form, enabling tractable optimization and leading to a simplified $Q$ function in the E-step of the algorithm. This technique allows us to reduce our formulation to a matrix sensing problem, using which we establish theoretical convergence guarantees for our algorithm. We conduct extensive experiments on real-world and synthetic datasets. These experiments demonstrate the advantages of using our algorithm over several baselines and confirm its strong robustness to both spammers and adversarial workers, highlighting its practical effectiveness in realistic crowdsourcing and reward learning settings. The code and data is publicly available at https://github.com/KaustubhShejole/BoRa_EM.

Problem

• pairwise comparisons
• worker reliability
• reward learning
• crowdsourcing
• spamming

Innovation

• Boltzmann-rational model
• Polya-Gamma augmentation
• EM algorithm
• matrix sensing
• worker reliability