Rich-Media Re-Ranker: A User Satisfaction-Driven LLM Re-ranking Framework for Rich-Media Search

2026-02-05

AI Summary

Existing re-ranking methods struggle to model users’ multidimensional intents and often neglect rich multimodal signals such as visual information, thereby limiting improvements in search satisfaction. To address this, this work proposes a user satisfaction–oriented re-ranking framework powered by large language models (LLMs). The approach employs a Query Planner to parse query evolution and disentangle user intents, integrates visual signals generated by a vision-language model (VLM), and leverages a multi-task reinforcement learning–enhanced LLM re-ranker to enable fine-grained, multidimensional relevance assessment. Extensive offline experiments demonstrate significant performance gains over state-of-the-art baselines, and the method has been successfully deployed in a large-scale industrial search system, yielding measurable improvements in online user engagement and satisfaction.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Re-ranking plays a crucial role in modern information search systems by refining the ranking of initial search results to better satisfy user information needs. However, existing methods show two notable limitations in improving user search satisfaction: inadequate modeling of multifaceted user intents and neglect of rich side information such as visual perception signals. To address these challenges, we propose the Rich-Media Re-Ranker framework, which aims to enhance user search satisfaction through multi-dimensional and fine-grained modeling. Our approach begins with a Query Planner that analyzes the sequence of query refinements within a session to capture genuine search intents, decomposing the query into clear and complementary sub-queries to enable broader coverage of users'potential intents. Subsequently, moving beyond primary text content, we integrate richer side information of candidate results, including signals modeling visual content generated by the VLM-based evaluator. These comprehensive signals are then processed alongside carefully designed re-ranking principle that considers multiple facets, including content relevance and quality, information gain, information novelty, and the visual presentation of cover images. Then, the LLM-based re-ranker performs the holistic evaluation based on these principles and integrated signals. To enhance the scenario adaptability of the VLM-based evaluator and the LLM-based re-ranker, we further enhance their capabilities through multi-task reinforcement learning. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art baselines. Notably, the proposed framework has been deployed in a large-scale industrial search system, yielding substantial improvements in online user engagement rates and satisfaction metrics.

Problem

• re-ranking
• user satisfaction
• rich-media search
• user intent
• visual perception

Innovation

• Rich-Media Re-Ranking
• User Intent Modeling
• Visual-Language Model (VLM)
• LLM-based Re-ranker
• Multi-task Reinforcement Learning