Balancing Embedding Spectrum for Recommendation

2024-06-17

arXiv.org

AI Summary

Embedding collapse in recommender systems causes representation degradation, manifesting as excessive concentration of user/item embeddings within a low-dimensional subspace—particularly prone under pairwise learning paradigms and leading to representation collapse. To address this, we propose DirectSpec, the first explicit all-pass filtering mechanism that enforces spectral balance of embedding distributions; we further introduce its self-adaptive gradient variant, DirectSpec+. We establish, for the first time, a theoretical connection between spectral balance and uniformity in contrastive learning. Our method integrates spectral analysis, embedding decorrelation optimization, and self-paced gradient updates, and is compatible with both Matrix Factorization (MF) and LightGCN architectures. Extensive experiments on multiple benchmark datasets demonstrate significant improvements over state-of-the-art methods: Recall@20 increases by 3.2%–5.7%. DirectSpec also achieves higher training efficiency than negative-sampling baselines, effectively mitigating embedding collapse and enhancing model capacity.

Technology Category

• Machine Learning: Representation Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Modern recommender systems heavily rely on high-quality representations learned from high-dimensional sparse data. While significant efforts have been invested in designing powerful algorithms for extracting user preferences, the factors contributing to good representations have remained relatively unexplored. In this work, we shed light on an issue in the existing pair-wise learning paradigm (i.e., the embedding collapse problem), that the representations tend to span a subspace of the whole embedding space, leading to a suboptimal solution and reducing the model capacity. Specifically, optimization on observed interactions is equivalent to a low pass filter causing users/items to have the same representations and resulting in a complete collapse. While negative sampling acts as a high pass filter to alleviate the collapse by balancing the embedding spectrum, its effectiveness is only limited to certain losses, which still leads to an incomplete collapse. To tackle this issue, we propose a novel method called DirectSpec, acting as a reliable all pass filter to balance the spectrum distribution of the embeddings during training, ensuring that users/items effectively span the entire embedding space. Additionally, we provide a thorough analysis of DirectSpec from a decorrelation perspective and propose an enhanced variant, DirectSpec+, which employs self-paced gradients to optimize irrelevant samples more effectively. Moreover, we establish a close connection between DirectSpec+ and uniformity, demonstrating that contrastive learning (CL) can alleviate the collapse issue by indirectly balancing the spectrum. Finally, we implement DirectSpec and DirectSpec+ on two popular recommender models: MF and LightGCN. Our experimental results demonstrate its effectiveness and efficiency over competitive baselines.

Problem

• Address embedding collapse in recommender systems
• Balance spectrum distribution of embeddings
• Enhance model capacity with DirectSpec method

Innovation

• DirectSpec balances embedding spectrum effectively
• DirectSpec+ optimizes samples with self-paced gradients
• Contrastive learning indirectly alleviates embedding collapse