The Benefits of Diversity: Combining Comparisons and Ratings for Efficient Scoring

2026-02-08

AI Summary

This work addresses the challenge of effectively integrating two distinct types of preference signals—individual ratings and pairwise comparisons—to improve the accuracy of inferred entity scores. The authors propose SCoRa, a unified probabilistic graphical model that jointly learns entity scores through maximum a posteriori (MAP) estimation. They provide the first systematic theoretical demonstration that combining both signal types yields significantly better performance than methods relying on either signal alone, particularly in accurately ranking top-tier entities. Theoretical analysis establishes the model’s monotonicity and robustness, while empirical results confirm that SCoRa reliably recovers true scores even under model misspecification and consistently outperforms baseline approaches that use only ratings or only pairwise comparisons in real-world scenarios.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Should humans be asked to evaluate entities individually or comparatively? This question has been the subject of long debates. In this work, we show that, interestingly, combining both forms of preference elicitation can outperform the focus on a single kind. More specifically, we introduce SCoRa (Scoring from Comparisons and Ratings), a unified probabilistic model that allows to learn from both signals. We prove that the MAP estimator of SCoRa is well-behaved. It verifies monotonicity and robustness guarantees. We then empirically show that SCoRa recovers accurate scores, even under model mismatch. Most interestingly, we identify a realistic setting where combining comparisons and ratings outperforms using either one alone, and when the accurate ordering of top entities is critical. Given the de facto availability of signals of multiple forms, SCoRa additionally offers a versatile foundation for preference learning.

Problem

• preference elicitation
• scoring
• comparisons
• ratings
• top-k ranking

Innovation

• preference learning
• probabilistic modeling
• comparison and rating fusion
• SCoRa
• robust scoring