Designing Rules to Pick a Rule: Aggregation by Consistency

2025-08-23

AI Summary

When multiple evaluators rank items, how can the optimal aggregation rule be automatically selected? Existing methods exhibit complementary strengths and weaknesses, and impossibility theorems preclude any rule from simultaneously satisfying all desirable axiomatic properties. Method: We propose a consistency-driven, data-adaptive framework for aggregation rule selection, optimizing for maximal pairwise (or triplet) consistency. Our approach introduces a novel axiomatization for rule selection and implements a sampling-based algorithm that operates without assuming an underlying data-generating model. Contribution: This work bridges the long-standing gap between social choice theory and statistical ranking aggregation. The proposed mechanism provides formal axiomatic guarantees while remaining computationally tractable. Empirical evaluation demonstrates that it accurately recovers the maximum-likelihood estimator under diverse statistical models—including Mallows, Plackett–Luce, and Thurstone—while significantly improving ranking consistency across synthetic benchmarks and real-world applications.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Game Theory and Economic Paradigms: Social Choice / Voting
• Multiagent Systems: Mechanism Design

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Given a set of items and a set of evaluators who all individually rank them, how do we aggregate these evaluations into a single societal ranking? Work in social choice and statistics has produced many aggregation methods for this problem, each with its desirable properties, but also with its limitations. Further, existing impossibility results rule out designing a single method that achieves every property of interest. Faced with this trade-off between incompatible desiderata, how do we decide which aggregation rule to use, i.e., what is a good rule picking rule? In this paper, we formally address this question by introducing a novel framework for rule picking rules (RPRs). We then design a data-driven RPR that identifies the best aggregation method for each specific setting, without assuming any generative model. The principle behind our RPR is to pick the rule which maximizes the consistency of the output ranking if the data collection process were repeated. We introduce several consistency-related axioms for RPRs and show that our method satisfies them, including those failed by a wide class of natural RPRs. While we prove that the algorithmic problem of maximizing consistency is computationally hard, we provide a sampling-based implementation of our RPR that is efficient in practice. We run this implementation on known statistical models and find that, when possible, our method selects the maximum likelihood estimator of the data. Finally, we show that our RPR can be used in many real-world settings to gain insights about how the rule currently being used can be modified or replaced to substantially improve the consistency of the process. Taken together, our work bridges an important gap between the axiomatic and statistical approaches to rank aggregation, laying a robust theoretical and computational foundation for principled rule picking.

Problem

• How to aggregate individual rankings into a societal ranking
• How to choose the best aggregation rule for specific settings
• Maximizing consistency in rank aggregation without generative models

Innovation

• Data-driven rule picking without generative models
• Maximizes consistency through repeat sampling approach
• Efficient sampling-based implementation for practical use