MVEB: Massive Video Embedding Benchmark

2026-06-12

AI Summary

Existing video embeddings lack a unified, diverse, and efficient multi-task evaluation benchmark. To address this gap, this work introduces MVEB, a large-scale benchmark for video embedding evaluation encompassing 23 task types—including classification, clustering, retrieval, and question answering—and integrated into the MTEB multimodal evaluation framework. The study presents the first systematic assessment of 33 models across a broad spectrum of video tasks, revealing that the impact of audio on performance critically depends on the annotation source. A cost-effective yet representative subset is also curated. Experimental results demonstrate no single model universally dominates: multimodal large language models (MLLMs) excel in classification and question answering, while multimodal alignment approaches outperform in retrieval and zero-shot classification. The project open-sources all 184 tasks and maintains a public leaderboard.

Abstract

We introduce the Massive Video Embedding Benchmark (MVEB), a 23-task benchmark for video embeddings spanning classification, zero-shot classification, clustering, pair classification, retrieval, and video-centric question answering. We evaluate 33 models and find that no single model dominates: MLLM-based embeddings lead on classification, clustering, pair classification, and QA; multimodal binding leads on retrieval and zero-shot classification; generative MLLMs without contrastive adaptation collapse on cross-modal tasks. Paired video-only vs. audio+video evaluations show that audio's contribution depends on dataset annotation provenance: audio helps when labels were produced from both modalities and hurts when they were produced from visuals alone, a six-point gap consistent across model families. MVEB is derived from MVEB+, a 184-task pool, and is designed to maintain task diversity while reducing evaluation cost. It integrates into the MTEB ecosystem for unified evaluation across text, image, audio, and video. We release MVEB and all 184 tasks along with code and a leaderboard at https://github.com/embeddings-benchmark/mteb.

Problem

• video embedding
• benchmark
• multimodal evaluation
• model comparison
• task diversity

Innovation

• video embedding
• multimodal benchmark
• MVEB
• zero-shot classification
• cross-modal evaluation