The Pitfall of Scaling Up: Uncovering and Mitigating Popularity Bias Amplification in Scaling Transformer-based Recommenders

2026-06-20

AI Summary

This work addresses the unintended consequence of scaling up Transformer-based sequential recommendation models: while larger models improve accuracy, they exacerbate popularity bias and harm long-tail fairness. The study reveals, for the first time, that this phenomenon stems from a synergistic effect between attention aggregation and feed-forward projection induced by increased model depth, leading to spectral collapse in predictions. To mitigate this, the authors propose SPRINT, a method that constrains the column sums of the attention score matrix and the spectral norm of the feed-forward layers during model scaling. Extensive experiments demonstrate that SPRINT consistently enhances both recommendation accuracy and long-tail fairness across model sizes ranging from 0.05M to 0.34B parameters, yielding healthier scaling behavior.

Abstract

We identify a critical pitfall in scaling transformer-based sequential recommenders: while increasing model size improves recommendation accuracy, it simultaneously amplifies popularity bias. This bias drives systems to over-recommend popular items at the expense of niche ones, which not only undermines fairness but also degrades the broader ecosystem by reinforcing the Matthew effect and filter bubbles. Consequently, this bias amplification emerges as a fundamental obstacle to sustainable model scaling. Through comprehensive theoretical and empirical analyses, we uncover the root cause of this amplification. Our findings reveal that as model depth increases, the two core components of the transformer architecture, i.e., attention aggregation and feed-forward projections, synergistically induce severe spectral collapse in model predictions, which directly translates to the amplification of popularity bias. To address this challenge, we propose SPRINT (Scalable Popularity Regularization IN Transformers), which mitigates spectral collapse during scaling by constraining (i) the maximum column-sums of the attention score matrices and (ii) the spectral norms of the feed-forward parameters. Extensive experiments demonstrate that SPRINT significantly improves both accuracy and long-tail fairness. Crucially, it yields more favorable scaling behaviors when expanding model sizes from 0.05M to 0.34B parameters. The code is available at https://github.com/Tiny-Snow/GenRec.

Problem

• popularity bias
• model scaling
• transformer-based recommenders
• spectral collapse
• long-tail fairness

Innovation

• popularity bias
• spectral collapse
• transformer scaling
• sequential recommendation
• fairness