PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs

2025-10-08

AI Summary

Current sentence embedding evaluations predominantly rely on static benchmarks (e.g., MTEB), leading to inflated estimates of model robustness. To address this, we propose PTEB—a novel, LLM-powered evaluation protocol that dynamically generates semantically preserved paraphrases via iterative, stochastic rewriting and statistical aggregation, enabling real-time, token-level assessment of embedding stability under lexical perturbations. Unlike static benchmarks, PTEB eliminates dependence on fixed test sets and incorporates semantic-similarity-guided, efficient paraphrase generation. We validate its effectiveness and statistical robustness across seven MTEB tasks and three multilingual datasets. Our experiments reveal, for the first time, that state-of-the-art embedding models exhibit high sensitivity to fine-grained token-level variations—and critically, this vulnerability shows no significant correlation with model scale.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Large-scale security measurements

Abstract

Current evaluations of sentence embedding models typically rely on static test beds such as the Massive Text Embedding Benchmark (MTEB). While invaluable, repeated tuning on a fixed suite can inflate reported performance and obscure real-world robustness. We introduce the Paraphrasing Text Embedding Benchmark (PTEB), a dynamic protocol that stochastically generates meaning-preserving paraphrases at evaluation time and aggregates results across multiple runs. Using a cost-efficient LLM-based method grounded in semantic textual similarity gold ratings, we show that LLMs generate token-diverse but semantically preserving, paraphrases. Across 7 MTEB tasks, we validate our hypothesis that the performance of sentence encoders is sensitive to changes in token space even when semantics remain fixed. We also observe that smaller models are not disproportionately affected relative to larger ones. Our results are statistically robust over multiple runs and we extended our experiments to 3 multilingual datasets covering 10 languages. More generally, we aim to propose a new evaluation paradigm in NLP that relies less on static, pre-defined benchmarks but shifts towards dynamic, stochastic evaluation leveraging eval-time compute.

Problem

• Evaluating sentence embedding robustness against semantic-preserving paraphrases
• Addressing performance inflation from static benchmark overfitting
• Proposing dynamic evaluation using stochastic paraphrasing at test time

Innovation

• Dynamic paraphrase generation using LLMs
• Stochastic evaluation protocol for robustness testing
• Semantic similarity-based multilingual benchmark validation