SHIFT: Semantic Harmonization via Index-side Feature Transformation for Multilingual Information Retrieval

2026-06-17

AI Summary

This work addresses the challenge in multilingual dense retrieval where language consistency bias between queries and documents hinders the recall of semantically relevant but cross-lingual document matches. To mitigate this issue, the authors propose a training-free, index-time debiasing method that leverages parallel translation corpora to estimate language-specific offset vectors relative to a source language. These offsets are then subtracted from document embeddings during indexing to achieve language-agnostic semantic alignment. This approach introduces, for the first time, an unsupervised language shift correction mechanism on the indexing side, effectively reducing language bias without requiring model fine-tuning. Experimental results demonstrate significant improvements in cross-lingual recall across four established multilingual retrieval benchmarks.

Abstract

With the rapid expansion of massive multilingual corpora, Multilingual Information Retrieval (MLIR) has emerged as a critical technology for global information access. MLIR enables users to retrieve semantically relevant documents from multilingual text collections using a single-language query. However, recent multilingual dense retrieval models often exhibit a strong preference for documents in the same language as the query. This leads to severe language bias, where top-ranked results are dominated by documents of specific languages, even when documents in other languages contain more semantically relevant information. To address this issue, we propose SHIFT, a training-free method applicable in the indexing stage. Specifically, SHIFT utilizes parallel translation pairs to estimate a relative language vector for each target language with respect to a source language. Subsequently, SHIFT corrects the language-specific offset by subtracting this relative language vector from document embeddings during indexing. Our comprehensive evaluation across four MLIR benchmarks and diverse dense retrieval models confirms that SHIFT can effectively mitigate language bias and enhance MLIR performance.

Problem

• Multilingual Information Retrieval
• language bias
• dense retrieval
• semantic relevance

Innovation

• language bias mitigation
• training-free method
• index-side feature transformation
• multilingual dense retrieval
• semantic harmonization