Q-BERT4Rec: Quantized Semantic-ID Representation Learning for Multimodal Recommendation

2025-12-02

AI Summary

Existing Transformer-based sequential recommendation methods (e.g., BERT4Rec) rely solely on discrete item IDs, neglecting rich multimodal semantic information—such as text and images—leading to poor generalization and limited interpretability. To address this, we propose the first semantic unification and quantization framework for multimodal sequential recommendation. Our approach: (i) introduces cross-modal semantic injection to map heterogeneous modalities (text, images) into a unified semantic ID space; (ii) pioneers residual vector quantization (RVQ) for semantic tokenization in sequential recommendation; and (iii) designs a multi-region masking pretraining strategy (span/tail/multi-region) to enhance sequential understanding and generalization. Extensive experiments on Amazon multimodal benchmarks demonstrate significant improvements over state-of-the-art methods—including BERT4Rec and CLIP4Rec—validating that semantic tokenization simultaneously boosts both recommendation accuracy and model interpretability.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Sequential recommendation plays a critical role in modern online platforms such as e-commerce, advertising, and content streaming, where accurately predicting users'next interactions is essential for personalization. Recent Transformer-based methods like BERT4Rec have shown strong modeling capability, yet they still rely on discrete item IDs that lack semantic meaning and ignore rich multimodal information (e.g., text and image). This leads to weak generalization and limited interpretability. To address these challenges, we propose Q-Bert4Rec, a multimodal sequential recommendation framework that unifies semantic representation and quantized modeling. Specifically, Q-Bert4Rec consists of three stages: (1) cross-modal semantic injection, which enriches randomly initialized ID embeddings through a dynamic transformer that fuses textual, visual, and structural features; (2) semantic quantization, which discretizes fused representations into meaningful tokens via residual vector quantization; and (3) multi-mask pretraining and fine-tuning, which leverage diverse masking strategies -- span, tail, and multi-region -- to improve sequential understanding. We validate our model on public Amazon benchmarks and demonstrate that Q-Bert4Rec significantly outperforms many strong existing methods, confirming the effectiveness of semantic tokenization for multimodal sequential recommendation. Our source code will be publicly available on GitHub after publishing.

Problem

• Enhances item embeddings with multimodal semantic information
• Replaces discrete IDs with quantized semantic tokens for recommendations
• Improves generalization and interpretability in sequential recommendation systems

Innovation

• Cross-modal semantic injection enriches ID embeddings with multimodal features.
• Semantic quantization discretizes representations via residual vector quantization.
• Multi-mask pretraining uses diverse strategies to enhance sequential understanding.