Progressive Semantic Residual Quantization for Multimodal-Joint Interest Modeling in Music Recommendation

2025-08-27

AI Summary

To address two key challenges in multimodal music recommendation—semantic degradation within modalities and modeling gaps across modalities—this paper proposes a two-stage joint modeling framework. First, we introduce Progressive Semantic Residual Quantization (PSRQ), which generates high-quality discrete IDs while preserving original semantics to mitigate semantic drift. Second, we design a Multi-Codebook Cross-Attention (MCCA) network that simultaneously captures modality-specific preferences and cross-modal correlations. This framework unifies fine-grained intra-modal representation learning with robust cross-modal fusion. Extensive experiments on multiple real-world datasets demonstrate significant improvements over state-of-the-art methods. Furthermore, the proposed approach has been deployed on a leading domestic music streaming platform; online A/B tests confirm substantial gains in core business metrics—including click-through rate and completion rate—validating both its effectiveness and industrial applicability.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

In music recommendation systems, multimodal interest learning is pivotal, which allows the model to capture nuanced preferences, including textual elements such as lyrics and various musical attributes such as different instruments and melodies. Recently, methods that incorporate multimodal content features through semantic IDs have achieved promising results. However, existing methods suffer from two critical limitations: 1) intra-modal semantic degradation, where residual-based quantization processes gradually decouple discrete IDs from original content semantics, leading to semantic drift; and 2) inter-modal modeling gaps, where traditional fusion strategies either overlook modal-specific details or fail to capture cross-modal correlations, hindering comprehensive user interest modeling. To address these challenges, we propose a novel multimodal recommendation framework with two stages. In the first stage, our Progressive Semantic Residual Quantization (PSRQ) method generates modal-specific and modal-joint semantic IDs by explicitly preserving the prefix semantic feature. In the second stage, to model multimodal interest of users, a Multi-Codebook Cross-Attention (MCCA) network is designed to enable the model to simultaneously capture modal-specific interests and perceive cross-modal correlations. Extensive experiments on multiple real-world datasets demonstrate that our framework outperforms state-of-the-art baselines. This framework has been deployed on one of China's largest music streaming platforms, and online A/B tests confirm significant improvements in commercial metrics, underscoring its practical value for industrial-scale recommendation systems.

Problem

• Addresses intra-modal semantic degradation in multimodal quantization
• Resolves inter-modal modeling gaps in user interest fusion
• Improves multimodal-joint interest modeling for music recommendation

Innovation

• Progressive Semantic Residual Quantization preserves prefix semantics
• Multi-Codebook Cross-Attention captures modal-specific interests
• Framework enables cross-modal correlation modeling simultaneously