Revisiting scalable sequential recommendation with Multi-Embedding Approach and Mixture-of-Experts

2025-10-29

AI Summary

To address scalability bottlenecks in sequential recommendation arising from modeling item multi-dimensional features and dynamic user-context relevance, this paper proposes Fuxi-MME—a novel framework that decouples a monolithic embedding into multiple low-dimensional, task-specific embeddings. It integrates a Mixture-of-Experts (MoE) mechanism with a lightweight Fuxi Block to enable feature-level specialized representation learning and dynamic gated fusion. Unlike conventional single-embedding + fully connected architectures, Fuxi-MME reduces the parameter growth order significantly while preserving high representational capacity, thereby improving both training and inference efficiency. Extensive experiments on multiple public sequential recommendation benchmarks demonstrate that Fuxi-MME consistently outperforms state-of-the-art baselines—including SASRec, BERT4Rec, and GRU4Rec—with substantially fewer parameters. This validates its dual advantages in recommendation accuracy and model scalability.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Recommender Systems

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In recommendation systems, how to effectively scale up recommendation models has been an essential research topic. While significant progress has been made in developing advanced and scalable architectures for sequential recommendation(SR) models, there are still challenges due to items' multi-faceted characteristics and dynamic item relevance in the user context. To address these issues, we propose Fuxi-MME, a framework that integrates a multi-embedding strategy with a Mixture-of-Experts (MoE) architecture. Specifically, to efficiently capture diverse item characteristics in a decoupled manner, we decompose the conventional single embedding matrix into several lower-dimensional embedding matrices. Additionally, by substituting relevant parameters in the Fuxi Block with an MoE layer, our model achieves adaptive and specialized transformation of the enriched representations. Empirical results on public datasets show that our proposed framework outperforms several competitive baselines.

Problem

• Scaling sequential recommendation models effectively
• Capturing items' multi-faceted characteristics dynamically
• Addressing dynamic item relevance in user context

Innovation

• Multi-embedding strategy captures diverse item characteristics
• Mixture-of-Experts architecture enables adaptive representation transformation
• Decomposed embedding matrices efficiently model multifaceted item properties