Efficient Elicitation of Collective Disagreements

2026-05-19

AI Summary

This study addresses the limitation of traditional survey methods—particularly those relying solely on pairwise comparisons—in distinguishing structural disagreement from random noise in group preferences. The authors propose a hierarchical framework that employs higher-order preference matrices to capture the topological structure of group choices over arbitrary subsets of alternatives. Through theoretical analysis, they demonstrate that most existing disagreement measures require preference information at least up to the third level (i.e., ternary subsets), as pairwise data alone are insufficient to reveal essential disagreement structures. To facilitate practical implementation, they design an efficient protocol for eliciting higher-order preference matrices that balances participant count and cognitive load. Empirical experiments validate the critical value of higher-order preference information for accurately quantifying disagreement and offer a low-burden, scalable approach to preference collection.

Abstract

We analyze the structure of the disagreement among a population of voters over a set of alternatives. Surveys typically ask either for pairwise comparisons, simple and intuitive for participants, or full rankings over alternatives, eliciting the entire voters' preferences. Building on the observation that pairwise comparisons cannot distinguish structural disagreement from noise, we propose a stratified framework to identify the minimal aggregated preference information needed to compute a number of disagreement measures from the literature. Specifically, we introduce the plurality matrix, a generalization of pairwise comparisons that records, for every subset $S$ of alternatives, the probability that each $a \in S$ ranks first in $S$. We define the level of a disagreement measure as the smallest subset size needed to express it, showing that many existing notions, including rank-variance and divisiveness, sit at level $3$, proving that pairwise comparisons are not enough. In addition, we demonstrate the interest of going beyond level $3$ both theoretically and experimentally. To make these results actionable, we design two elicitation protocols to estimate the plurality matrix, exploring the trade-off between the number of required participants and the cognitive load requested to each of them.

Problem

• collective disagreement
• preference elicitation
• pairwise comparisons
• plurality matrix
• disagreement measures

Innovation

• plurality matrix
• disagreement measurement
• preference elicitation
• stratified framework
• collective disagreement