ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs

2026-06-30

AI Summary

Current benchmarks for evaluating text embeddings suffer from limitations including static design, restricted language coverage, susceptibility to overfitting, and inadequate support for low-resource languages. This work proposes a cross-lingual, paragraph-level embedding evaluation framework grounded in Abstract Meaning Representation (AMR). By generating English minimal pairs with controlled semantic perturbations and pairing them with translations into target languages, the framework enables fine-grained, robust, and dynamically extensible semantic evaluation. Requiring only English–target parallel corpora, it supports assessment for any language. Experiments across more than 275 languages reveal substantial performance disparities and representational deficiencies in existing multilingual embedding models, particularly for low-resource languages and specific semantic phenomena.

Abstract

Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge. Current benchmarks are static, cover only a limited set of languages, are often domain-specific, susceptible to overfitting, and poorly representative of low-resource languages. To address these limitations, we introduce ALEE, a framework that extends Sentence Smith (Li et al., 2025) to the cross-lingual and paragraph level. ALEE uses Abstract Meaning Representations (AMR) to generate English minimal pairs with controlled, fine-grained semantic shifts, which are paired with translations in target languages. This approach enables targeted diagnostics for models in any language with English parallel data. We conduct a large-scale empirical study across a diverse set of embedding models and 275+ languages spanning three parallel datasets. On ALEE, performance varies substantially across languages, text lengths, and linguistic phenomena, exposing persistent gaps in cross-lingual semantic representation that track language prevalence in training resources and subword tokenization. We release ALEE at https://github.com/Andrian0s/any-lang-embed-eval

Problem

• text embeddings
• semantic similarity
• cross-lingual evaluation
• low-resource languages
• benchmark limitations

Innovation

• cross-lingual evaluation
• minimal pairs
• Abstract Meaning Representation
• text embeddings
• low-resource languages