UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning

2025-10-15

AI Summary

Existing general-purpose multimodal embedding models suffer from limited fine-grained semantic discrimination, ineffective hard negative mining, and poor identification of false negatives. To address these issues, we propose UniME-V2, which introduces an innovative MLLM-as-a-Judge mechanism: a large language model evaluates query-candidate pairs for fine-grained semantic alignment and generates soft matching scores, enabling high-quality hard negative selection and false negative mitigation. Methodologically, UniME-V2 integrates global retrieval to construct a diverse negative pool, soft-label supervision, and similarity matrix alignment. Furthermore, we design the UniME-V2-Reranker—a unified re-ranking module supporting joint pairwise and listwise optimization. Evaluated on the MMEB benchmark and multiple cross-modal retrieval tasks, UniME-V2 achieves state-of-the-art performance, significantly enhancing discriminative capability and generalization under complex semantic scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Universal multimodal embedding models are foundational to various tasks. Existing approaches typically employ in-batch negative mining by measuring the similarity of query-candidate pairs. However, these methods often struggle to capture subtle semantic differences among candidates and lack diversity in negative samples. Moreover, the embeddings exhibit limited discriminative ability in distinguishing false and hard negatives. In this paper, we leverage the advanced understanding capabilities of MLLMs to enhance representation learning and present a novel Universal Multimodal Embedding (UniME-V2) model. Our approach first constructs a potential hard negative set through global retrieval. We then introduce the MLLM-as-a-Judge mechanism, which utilizes MLLMs to assess the semantic alignment of query-candidate pairs and generate soft semantic matching scores. These scores serve as a foundation for hard negative mining, mitigating the impact of false negatives and enabling the identification of diverse, high-quality hard negatives. Furthermore, the semantic matching scores are used as soft labels to mitigate the rigid one-to-one mapping constraint. By aligning the similarity matrix with the soft semantic matching score matrix, the model learns semantic distinctions among candidates, significantly enhancing its discriminative capacity. To further improve performance, we propose UniME-V2-Reranker, a reranking model trained on our mined hard negatives through a joint pairwise and listwise optimization approach. We conduct comprehensive experiments on the MMEB benchmark and multiple retrieval tasks, demonstrating that our method achieves state-of-the-art performance on average across all tasks.

Problem

• Captures subtle semantic differences among multimodal candidate embeddings
• Enhances discriminative ability for false and hard negative samples
• Improves diversity and quality of negative samples in retrieval

Innovation

• Uses MLLM-as-a-Judge for semantic alignment assessment
• Employs soft semantic scores for hard negative mining
• Aligns similarity matrix with soft labels for discrimination