Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks

2025-06-26

AI Summary

To address the longstanding issues of limited longevity and poor reproducibility in large-scale text embedding benchmarks (MTEB), this work proposes a systematic engineering framework. It introduces a continuous integration (CI)-driven automated pipeline incorporating data integrity validation, task-level regression testing, and cross-task result generalizability assessment. The framework features a modular, extensible benchmark architecture enabling low-barrier integration of new tasks and datasets. Crucially, it pioneers the adoption of software engineering best practices—such as versioned data snapshots and test-driven benchmark development—into embedding evaluation. Experiments demonstrate that the framework scales MTEB to over 100 tasks and 80+ datasets while ensuring consistent, reproducible evaluation outcomes across diverse models and execution environments. As a result, it has become the de facto standard evaluation platform for state-of-the-art embedding research.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

The Massive Text Embedding Benchmark (MTEB) has become a standard evaluation platform for text embedding models. While previous work has established the core benchmark methodology, this paper focuses on the engineering aspects that ensure MTEB's continued reproducibility and extensibility. We present our approach to maintaining robust continuous integration pipelines that validate dataset integrity, automate test execution, and assess benchmark results' generalizability. We detail the design choices that collectively enhance reproducibility and usability. Furthermore, we discuss our strategies for handling community contributions and extending the benchmark with new tasks and datasets. These engineering practices have been instrumental in scaling MTEB to become more comprehensive while maintaining quality and, ultimately, relevance to the field. Our experiences offer valuable insights for benchmark maintainers facing similar challenges in ensuring reproducibility and usability in machine learning evaluation frameworks. The MTEB repository is available at: https://github.com/embeddings-benchmark/mteb

Problem

• Ensuring long-term reproducibility of MTEB benchmarks
• Maintaining robust continuous integration for dataset integrity
• Scaling MTEB with community contributions and new tasks

Innovation

• Robust continuous integration pipelines
• Enhanced reproducibility and usability design
• Community contributions and benchmark extension