GME: Improving Universal Multimodal Retrieval by Multimodal LLMs

2024-12-22

arXiv.org

AI Summary

Existing universal multimodal retrieval (UMR) approaches suffer from severe modality imbalance in training data, absence of high-quality unified training corpora, and lack of dedicated evaluation benchmarks. Method: We propose GME, an MLLM-based dense retriever, featuring a modality-balanced synthetic data pipeline to construct UMRB—the first comprehensive, UMR-specific benchmark—and an end-to-end trainable, multimodally aligned dense retrieval framework integrating contrastive learning, instruction tuning, and modality-aligned embedding. Contribution/Results: This work presents the first MLLM-driven dense retrieval solution for UMR. We publicly release both the UMRB benchmark and a high-quality synthetic dataset. Extensive experiments on UMRB demonstrate that GME consistently outperforms state-of-the-art methods, empirically validating the synergistic gains of model scale, training strategy, and data quality on cross-modal retrieval performance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Universal Multimodal Retrieval (UMR) aims to enable search across various modalities using a unified model, where queries and candidates can consist of pure text, images, or a combination of both. Previous work has attempted to adopt multimodal large language models (MLLMs) to realize UMR using only text data. However, our preliminary experiments demonstrate that more diverse multimodal training data can further unlock the potential of MLLMs. Despite its effectiveness, the existing multimodal training data is highly imbalanced in terms of modality, which motivates us to develop a training data synthesis pipeline and construct a large-scale, high-quality fused-modal training dataset. Based on the synthetic training data, we develop the General Multimodal Embedder (GME), an MLLM-based dense retriever designed for UMR. Furthermore, we construct a comprehensive UMR Benchmark (UMRB) to evaluate the effectiveness of our approach. Experimental results show that our method achieves state-of-the-art performance among existing UMR methods. Last, we provide in-depth analyses of model scaling and training strategies, and perform ablation studies on both the model and synthetic data.

Problem

• Enabling cross-modal search with a unified model
• Addressing imbalance in multimodal training data
• Developing MLLM-based dense retriever for UMR

Innovation

• Synthesizes diverse multimodal training data
• Develops MLLM-based dense retriever GME
• Constructs comprehensive UMR Benchmark UMRB