Molar: Multimodal LLMs with Collaborative Filtering Alignment for Enhanced Sequential Recommendation

2024-12-24

AI Summary

Existing sequential recommendation methods integrating large language models (LLMs) commonly neglect collaborative filtering signals and lack a unified semantic modeling capability for multimodal content (e.g., text, images, videos), leading to representation bias and insufficient recommendation coherence. To address this, we propose a post-alignment framework that jointly leverages collaborative filtering and multimodal large language models (MLLMs). Specifically, we first employ an MLLM to fuse item IDs, textual, and non-textual modalities into unified item representations. Subsequently, a cross-modal alignment mechanism jointly optimizes user ID–based implicit preferences with multimodal explicit semantics. This work constitutes the first effort to unify ID-driven implicit signals and multimodal semantic understanding within sequential recommendation. Extensive experiments on multiple public benchmarks demonstrate substantial improvements—up to 23.6% gain in Recall@10—over both conventional and LLM-based baselines, validating the efficacy of our joint modeling approach.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Sequential recommendation (SR) systems have evolved significantly over the past decade, transitioning from traditional collaborative filtering to deep learning approaches and, more recently, to large language models (LLMs). While the adoption of LLMs has driven substantial advancements, these models inherently lack collaborative filtering information, relying primarily on textual content data neglecting other modalities and thus failing to achieve optimal recommendation performance. To address this limitation, we propose Molar, a Multimodal large language sequential recommendation framework that integrates multiple content modalities with ID information to capture collaborative signals effectively. Molar employs an MLLM to generate unified item representations from both textual and non-textual data, facilitating comprehensive multimodal modeling and enriching item embeddings. Additionally, it incorporates collaborative filtering signals through a post-alignment mechanism, which aligns user representations from content-based and ID-based models, ensuring precise personalization and robust performance. By seamlessly combining multimodal content with collaborative filtering insights, Molar captures both user interests and contextual semantics, leading to superior recommendation accuracy. Extensive experiments validate that Molar significantly outperforms traditional and LLM-based baselines, highlighting its strength in utilizing multimodal data and collaborative signals for sequential recommendation tasks. The source code is available at https://anonymous.4open.science/r/Molar-8B06/.

Problem

• Sequence Recommendation
• Large Language Models
• Multimodal Information Integration

Innovation

• Molar Recommendation System
• Multimodal Information Integration
• Enhanced Collaborative Filtering