The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness

2026-05-27

AI Summary

This study addresses the limitations of existing text embedding benchmarks—such as MTEB—that rely on static scores and fail to capture the multidimensional fragility of models under real-world perturbations. The authors propose HTEB, a dynamic evaluation framework that conceptualizes embedding robustness as a multidimensional, dynamic property. HTEB leverages large language models to stochastically transform inputs along three interpretable dimensions during evaluation: lexical/style variation, text length, and language. These transformations are combined with human validation to establish a dynamic benchmarking paradigm. Experiments across 16 open-source embedding models and 32 datasets reveal that robustness across dimensions is largely decoupled, that scaling model size does not consistently narrow the performance gap between original and perturbed inputs, and that English data exhibits particular sensitivity to perturbations. This work thus transcends the constraints of conventional static benchmarks.

Abstract

Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that embedding robustness is multidimensional, since models respond differently to different types of variation, and requires dynamic evaluation to expose failures hidden by static benchmarks. We introduce the Harder Text Embedding Benchmark (HTEB), a dynamic evaluation framework that challenges model robustness along three practically interpretable axes (Lexical/Stylistic, Length and Language) by stochastically transforming inputs at evaluation time with an LLM. Evaluating 16 open-weight embedding models on 32 datasets covering 42 languages under transformations validated by 4,800 human ratings on an English subsample, we find three patterns: (1) Models exhibit specific, partly decoupled robustness profiles across axes. (2) Across three model families, scale increases absolute scores but does not close the gap between original and transformed evaluations. Here, scaling tends to improve specifically the Language axis. (3) English datasets are more sensitive to HTEB transformations than multilingual datasets. This demonstrates that HTEB identifies strengths and weaknesses of models along deployment-relevant axes, challenging current embedding benchmarks and arguing for multidimensional, dynamic robustness evaluation.

Problem

• text embedding
• robustness
• benchmark
• multidimensional evaluation
• dynamic evaluation

Innovation

• dynamic evaluation
• multidimensional robustness
• text embedding benchmark
• LLM-based transformation
• embedding robustness