HUME: Measuring the Human-Model Performance Gap in Text Embedding Task

2025-10-11

AI Summary

Current text embedding evaluation lacks reliable human performance baselines, hindering model interpretability and fair cross-lingual or cross-task comparisons. To address this, we propose HUME—the first scalable, multilingual, multitask human benchmark framework for text embeddings. HUME systematically evaluates human performance across 16 datasets from the MTEB suite, spanning re-ranking, classification, clustering, and semantic similarity tasks. Experiments span high- and low-resource languages, revealing a human average score of 77.6%, compared to 80.1% for state-of-the-art embedding models. While performance approaches human-level on several tasks, substantial gaps persist—particularly in low-resource languages and fine-grained semantic understanding. HUME is the first to quantitatively characterize the human-model capability gap, exposing critical limitations of prevailing embedding models. By anchoring evaluation to empirically measured human performance, HUME significantly enhances interpretability and completeness of embedding benchmarks.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Comparing human and model performance offers a valuable perspective for understanding the strengths and limitations of embedding models, highlighting where they succeed and where they fail to capture meaning and nuance. However, such comparisons are rarely made, as human performance on embedding tasks is difficult to measure. To fill this gap, we introduce HUME: Human Evaluation Framework for Text Embeddings. While frameworks like MTEB provide broad model evaluation, they lack reliable estimates of human performance, limiting the interpretability of model scores. We measure human performance across 16 MTEB datasets spanning reranking, classification, clustering, and semantic textual similarity across linguistically diverse high- and low-resource languages. Humans achieve an average performance of 77.6% compared to 80.1% for the best embedding model, although variation is substantial: models reach near-ceiling performance on some datasets while struggling on others, suggesting dataset issues and revealing shortcomings in low-resource languages. We provide human performance baselines, insight into task difficulty patterns, and an extensible evaluation framework that enables a more meaningful interpretation of the model and informs the development of both models and benchmarks. Our code, dataset, and leaderboard are publicly available at https://github.com/embeddings-benchmark/mteb.

Problem

• Measuring human-model performance gap in text embeddings
• Providing human baselines for embedding model evaluation
• Addressing dataset limitations in low-resource language embeddings

Innovation

• Introduces HUME framework for human evaluation
• Measures human performance across 16 MTEB datasets
• Provides human baselines for embedding model comparison