MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models

2026-04-25

AI Summary

This work addresses the absence of a systematic evaluation benchmark for existing omnimodal embedding models, which hinders accurate assessment of their semantic alignment and cross-modal retrieval capabilities. To this end, we introduce MMEB-V3—the first comprehensive benchmark encompassing text, images, videos, audio, and agent-based scenarios—and construct OmniSET, a dataset of semantically equivalent tuples designed to disentangle true semantic similarity from modality-specific artifacts. Leveraging this framework, we uncover three critical issues for the first time: query modality bias, target modality mismatch, and failure of instruction-guided retrieval. Our experiments reveal that state-of-the-art models exhibit significant deficiencies in adhering to modality constraints and achieving symmetric cross-modal retrieval, thereby providing a diagnostic foundation and clear directions for future research in omnimodal representation learning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space. However, existing methods and benchmarks remain largely limited to partial modality coverage, making it difficult to systematically evaluate full-modality representation learning. In this work, we take a step toward the full-modality setting. We introduce MMEB-V3, a comprehensive benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios. To enable more fine-grained diagnosis, we further construct OmniSET (Omni-modality Semantic Equivalence Tuples), where semantically equivalent instances are represented across modalities, allowing us to disentangle semantic similarity from modality effects. Through experiments on MMEB-V3, we conduct a systematic analysis of full-modality embeddings and identify three key findings: (1) models often fail to retrieve the intended target modality; (2) cross-modal retrieval is highly asymmetric and dominated by query-modality bias; and (3) instruction-induced shifts are either insufficient or misaligned with the target modality, and therefore do not reliably improve retrieval. These results indicate that current multimodal embeddings are not yet capable of reliably enforcing modality constraints specified by instructions, and consequently fail to exhibit consistent modality-aware retrieval behavior. We hope MMEB-V3 provides a useful benchmark for understanding and diagnosing these limitations, and for guiding future research on full-modality embeddings.

Problem

• multimodal embedding
• full-modality
• cross-modal retrieval
• modality-awareness
• semantic representation

Innovation

• full-modality embedding
• multimodal benchmark
• semantic equivalence
• cross-modal retrieval
• modality-awareness