Are We Really Making Progress in Group Recommendation? Unmasking the Tie-Breaking Illusion

2026-08-11

AI Summary

This work demonstrates that performance gains reported by existing group recommendation methods on standard benchmarks may stem from evaluation biases rather than genuine modeling advances. It systematically uncovers a previously overlooked source of systematic bias arising from the interaction between score compression during training and deterministic tie-breaking during evaluation. To address this, the paper introduces a tie-aware evaluation protocol that computes expected metrics via uniform random tie-breaking and incorporates temperature-scaled BPR to mitigate tie inflation. Reproducing multiple representative methods on the CAMRa2011 and Mafengwo datasets reveals that most previously reported improvements substantially diminish under the new protocol, with significant shifts in method rankings—highlighting that current evaluation practices are substantially misleading.

Abstract

Recent group recommendation methods have reported strong improvements on standard benchmarks, but it remains unclear whether these gains always reflect genuine advances in modeling group preferences. In this paper, we show that several recent methods are affected by a systematic evaluation bias caused by the interaction between training-time score compression and evaluation-time deterministic tie-breaking. Specifically, an additional sigmoid transformation before the BPR objective can greatly increase tied top scores, making top-K metrics such as HR@K and NDCG@K highly sensitive to how ties are resolved. We revisit recent representative methods and their baselines on CAMRa2011 and Mafengwo under both group and user recommendation settings, and evaluate them with a tie-aware protocol that computes the exact expectation of HR@K and NDCG@K under uniform random tie-breaking. Our results show that many previously reported improvements shrink substantially under tie-aware evaluation, and the relative ranking of methods can change markedly. We further show that the additional sigmoid may act as implicit margin smoothing during optimization, and that temperature-scaled BPR can retain much of this benefit without inducing severe tie inflation. Overall, our findings highlight the importance of tie-aware evaluation for establishing reliable progress in group recommendation. The code is available at https://github.com/songduoma/TieAwareGroupRec.

Problem

• group recommendation
• evaluation bias
• tie-breaking
• top-K metrics
• score ties

Innovation

• tie-aware evaluation
• group recommendation
• evaluation bias
• BPR loss
• score compression