MMTEB: Massive Multilingual Text Embedding Benchmark

2025-02-19

AI Summary

Existing text embedding evaluations suffer from limited task scale and narrow coverage of languages and domains. This work introduces MTEB-250+, the first ultra-large-scale multilingual benchmark, encompassing 250+ languages and 500+ high-quality tasks—including emerging challenges such as instruction-following, long-document retrieval, and code retrieval. To address scalability and reliability, we propose two innovations: (1) a task correlation–based downsampling strategy that drastically reduces computational overhead; and (2) adaptive hard-negative sampling coupled with a zero-shot English sub-benchmark to ensure both efficiency and robust ranking fidelity. Experiments demonstrate that MTEB-250+ reduces evaluation cost to approximately one-tenth of the full-scale benchmark while maintaining a Pearson correlation of ≥0.98 for model rankings. Empirical validation identifies multilingual-e5-large-instruct as the current state-of-the-art open-source embedding model.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more comprehensive evaluation, we introduce the Massive Multilingual Text Embedding Benchmark (MMTEB) - a large-scale, community-driven expansion of MTEB, covering over 500 quality-controlled evaluation tasks across 250+ languages. MMTEB includes a diverse set of challenging, novel tasks such as instruction following, long-document retrieval, and code retrieval, representing the largest multilingual collection of evaluation tasks for embedding models to date. Using this collection, we develop several highly multilingual benchmarks, which we use to evaluate a representative set of models. We find that while large language models (LLMs) with billions of parameters can achieve state-of-the-art performance on certain language subsets and task categories, the best-performing publicly available model is multilingual-e5-large-instruct with only 560 million parameters. To facilitate accessibility and reduce computational cost, we introduce a novel downsampling method based on inter-task correlation, ensuring a diverse selection while preserving relative model rankings. Furthermore, we optimize tasks such as retrieval by sampling hard negatives, creating smaller but effective splits. These optimizations allow us to introduce benchmarks that drastically reduce computational demands. For instance, our newly introduced zero-shot English benchmark maintains a ranking order similar to the full-scale version but at a fraction of the computational cost.

Problem

• Evaluates multilingual text embedding models comprehensively
• Introduces large-scale, diverse evaluation tasks
• Reduces computational cost with optimized methods

Innovation

• Massive Multilingual Text Embedding Benchmark
• Novel downsampling method for efficiency
• Optimized retrieval with hard negatives