Granular feedback merits sophisticated aggregation

2025-07-16

AI Summary

This paper addresses the challenge of accurately predicting population-level feedback distributions under limited individual-level feedback samples, with particular attention to modeling difficulties arising from increased feedback granularity. To overcome the insufficient accuracy of conventional regularized averaging methods for fine-grained feedback, we propose a feedback-granularity-adaptive dynamic aggregation framework. Our key finding is that feedback granularity critically modulates the efficacy of sophisticated aggregation methods: on a five-point Likert scale, the proposed method achieves comparable prediction accuracy using only ~50% of the sample size required by traditional approaches; in contrast, no significant improvement is observed for binary feedback. Empirical validation on multi-wave social attitude survey data demonstrates substantial reductions in data collection costs. The framework thus provides both theoretical grounding and a practical tool for high-resolution modeling of collective opinions.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Sentiment analysis and opinion mining

Abstract

Human feedback is increasingly used across diverse applications like training AI models, developing recommender systems, and measuring public opinion -- with granular feedback often being preferred over binary feedback for its greater informativeness. While it is easy to accurately estimate a population's distribution of feedback given feedback from a large number of individuals, cost constraints typically necessitate using smaller groups. A simple method to approximate the population distribution is regularized averaging: compute the empirical distribution and regularize it toward a prior. Can we do better? As we will discuss, the answer to this question depends on feedback granularity. Suppose one wants to predict a population's distribution of feedback using feedback from a limited number of individuals. We show that, as feedback granularity increases, one can substantially improve upon predictions of regularized averaging by combining individuals' feedback in ways more sophisticated than regularized averaging. Our empirical analysis using questions on social attitudes confirms this pattern. In particular, with binary feedback, sophistication barely reduces the number of individuals required to attain a fixed level of performance. By contrast, with five-point feedback, sophisticated methods match the performance of regularized averaging with about half as many individuals.

Problem

• Improving granular feedback aggregation methods
• Optimizing feedback prediction with limited samples
• Enhancing accuracy in multi-point feedback systems

Innovation

• Sophisticated aggregation improves granular feedback estimation
• Regularized averaging outperformed by advanced methods
• Five-point feedback halves required sample size