Serendipitous Recommendation with Multimodal LLM

2025-06-09

AI Summary

To address the lack of novelty and serendipity in short-video platform recommendation systems, this paper proposes an MLLM-driven hierarchical serendipitous recommendation framework. It employs a fine-tuned multimodal large language model (MLLM) to capture high-level semantic user interests and leverages chain-of-thought (CoT) reasoning to uncover latent, unexplored interest clusters from video content, thereby generating interpretable serendipity-guiding signals. These signals are then integrated with lightweight conventional models to achieve high-recall, low-latency serendipitous recommendations. This work pioneers a collaborative hierarchical paradigm that synergistically combines MLLMs with traditional recommenders, incorporating video understanding, cross-modal alignment, and prompt engineering. Online A/B testing demonstrates a 23.7% increase in serendipity rate and an 11.2% improvement in average user session duration. The framework has been deployed on an industrial-scale platform with over one billion daily active users.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Video Understanding & Activity Analysis
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Conventional recommendation systems succeed in identifying relevant content but often fail to provide users with surprising or novel items. Multimodal Large Language Models (MLLMs) possess the world knowledge and multimodal understanding needed for serendipity, but their integration into billion-item-scale platforms presents significant challenges. In this paper, we propose a novel hierarchical framework where fine-tuned MLLMs provide high-level guidance to conventional recommendation models, steering them towards more serendipitous suggestions. This approach leverages MLLM strengths in understanding multimodal content and user interests while retaining the efficiency of traditional models for item-level recommendation. This mitigates the complexity of applying MLLMs directly to vast action spaces. We also demonstrate a chain-of-thought strategy enabling MLLMs to discover novel user interests by first understanding video content and then identifying relevant yet unexplored interest clusters. Through live experiments within a commercial short-form video platform serving billions of users, we show that our MLLM-powered approach significantly improves both recommendation serendipity and user satisfaction.

Problem

• Enhancing recommendation serendipity with multimodal LLMs
• Integrating MLLMs into large-scale recommendation systems efficiently
• Discovering novel user interests through multimodal content understanding

Innovation

• Hierarchical framework combines MLLMs with traditional models
• Chain-of-thought strategy discovers novel user interests
• MLLM guidance enhances serendipity in recommendations