ELVA: Exploring Ranking-Driven Universal Multimodal Retrieval

2026-06-18

AI Summary

This work addresses the "granularity blindness" problem in contrastive learning for general-purpose multimodal retrieval, where negative samples of varying granularity are treated indistinguishably. To resolve this, the authors propose ELVA, a novel framework that introduces verifiable reward-based reinforcement learning (RLVR) into retrieval tasks for the first time. ELVA employs a rule-driven reward mechanism to optimize the ranking of negative samples without explicit supervision, thereby enlarging the similarity gap between positive and negative pairs and enhancing the model’s sensitivity to queries of diverse granularities. The study also introduces MRBench, the first benchmark tailored for evaluating multimodal retrieval under multi-granularity queries. Experiments demonstrate that ELVA achieves state-of-the-art performance on standard retrieval benchmarks and yields a significant 13.1% improvement on MRBench, effectively mitigating the granularity blindness issue.

Abstract

Leveraging Multimodal Large Language Models (MLLMs) via contrastive learning has become a mainstream paradigm for improving the performance of Universal Multimodal Retrieval (UMR). However, previous works have ignored the grain blindness when adapting the contrastive paradigm into retrieval tasks. Grain blindness refers to the tendency of the model to overlook grain-level information contained in the query, which is crucial for effectively handling complex queries. This stems from contrastive learning treating samples as a binary classification (positive/negative), while ignoring the different information carried by each negative sample. To address this, we argue that negatives should be treated differently according to their similarity to the positive sample, enabling the model to learn distinct grain information from each negative. In this paper, we introduce a simple but effective framework, called ELVA, a novel rule-based RL framework that mitigates grain blindness through ranking-driven MLLMs. 1) Instead of relying on reward models, we extend Reinforcement Learning with Verifiable Rewards (RLVR) to retrieval tasks, allowing the model to explore new ranking behaviors without explicit ranking labels. 2) By utilizing rule-based rewards, our approach jointly optimizes the ranking of negative samples while enlarging the similarity gap between positive and negative. To more precisely measure grain blindness, we further introduce MRBench, a new benchmark specifically designed for multi-grain query scenarios. ELVA achieves state-of-the-art results across standard retrieval benchmarks, and its notable 13.1% improvement on MRBench further demonstrates its effectiveness in alleviating grain blindness.

Problem

• grain blindness
• universal multimodal retrieval
• contrastive learning
• multimodal large language models
• negative sample ranking

Innovation

• grain blindness
• ranking-driven retrieval
• multimodal large language models
• reinforcement learning with verifiable rewards
• universal multimodal retrieval