Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders

2026-07-04

AI Summary

This study addresses the longstanding underrepresentation of Portuguese in text embedding evaluation, which has hindered the accurate assessment of multilingual models' performance on this language. To bridge this gap, the authors introduce MTEB-PT, the first comprehensive benchmark for Portuguese embeddings, comprising 14 datasets spanning semantic textual similarity, classification, retrieval, and reranking tasks. Under a unified evaluation protocol, they assess 17 embedding models and demonstrate that rankings derived from multilingual benchmarks do not reliably predict Portuguese-specific performance. Through language-specific fine-tuning using contrastive learning and Matryoshka representation learning, they achieve substantial gains in semantic textual similarity and retrieval tasks, with models retaining strong competitiveness even at lower embedding dimensions. The work releases the MTEB-PT benchmark, fine-tuned models, and full implementation code to support future research.

Abstract

Portuguese remains underrepresented in text embedding evaluation, despite being one of the most widely spoken languages in the world. As a result, embedding models are often selected based on English or multilingual metrics, while their effectiveness in Portuguese remains unclear. We present MTEB-PT, a Portuguese benchmark constructed from a subset of MMTEB, comprising 14 existing datasets across Semantic Textual Similarity (STS), classification, retrieval, and reranking. We use this benchmark to evaluate 17 open- and closed-source embedding models under a unified protocol. Our results show that Portuguese performance is strongly task-dependent: multilingual rankings do not reliably predict Portuguese-specific performance across task families, no single model dominates all settings, and models with stronger long-context capacity are particularly advantageous on longer-input tasks such as retrieval and reranking. The benchmark also shows that language-specific fine-tuning still improves model performance in Portuguese, especially on task types that match the adaptation data most closely. To examine this effect, we fine-tune three representative backbone models with Portuguese contrastive supervision and Matryoshka Representation Learning (MRL). These benchmark-informed baselines yield their strongest gains on STS, consistent with the predominantly symmetric supervision used during training, while also improving retrieval and remaining competitive under dimensional truncation. We release the MTEB-PT benchmark, the fine-tuned models, and the training and evaluation code.

Problem

• Portuguese text embeddings
• embedding evaluation
• language underrepresentation
• multilingual benchmarks
• sentence encoders

Innovation

• MTEB-PT
• Portuguese sentence encoders
• contrastive learning
• Matryoshka Representation Learning
• multilingual embedding evaluation