🤖 AI Summary
Multilingual pre-trained models (e.g., mBERT, XLM-R) lack task-specific optimization for cross-lingual dense retrieval. Method: This paper proposes a sentence-sequence modeling framework tailored for cross-lingual retrieval, introducing the first cross-lingually universal sentence-order consistency modeling mechanism. It jointly leverages masked sentence prediction and hierarchical contrastive loss to learn document-level multilingual shared representations—without relying on bilingual dictionaries or parallel sentence pairs for fine-tuning, but instead exploiting naturally occurring sentence-level sequential alignment in parallel documents. Contribution/Results: The method achieves significant improvements over mBERT, XLM-R, and state-of-the-art supervised and unsupervised approaches on four major cross-lingual retrieval benchmarks (BUCC, Tatoeba, MLDR, XCED), with average Recall@1 gains of 4.2–9.8 percentage points, empirically validating that sequence-structural priors substantially enhance cross-lingual representation learning.
📝 Abstract
Recently multi-lingual pre-trained language models (PLM) such as mBERT and XLM-R have achieved impressive strides in cross-lingual dense retrieval. Despite its successes, they are general-purpose PLM while the multilingual PLM tailored for cross-lingual retrieval is still unexplored. Motivated by an observation that the sentences in parallel documents are approximately in the same order, which is universal across languages, we propose to model this sequential sentence relation to facilitate cross-lingual representation learning. Specifically, we propose a multilingual PLM called masked sentence model (MSM), which consists of a sentence encoder to generate the sentence representations, and a document encoder applied to a sequence of sentence vectors from a document. The document encoder is shared for all languages to model the universal sequential sentence relation across languages. To train the model, we propose a masked sentence prediction task, which masks and predicts the sentence vector via a hierarchical contrastive loss with sampled negatives. Comprehensive experiments on four cross-lingual retrieval tasks show MSM significantly outperforms existing advanced pre-training models, demonstrating the effectiveness and stronger cross-lingual retrieval capabilities of our approach. Code and model will be available.
