Adapting Multilingual Embedding Models to Turkish via Cross-Lingual Tokenizer Surgery and Offline Distillation

2026-05-28

AI Summary

This study addresses the limitations of existing Turkish sentence embedding models—namely, restricted contextual capacity, suboptimal performance, and high training costs—by proposing an efficient three-stage adaptation framework. The approach begins with optimizing a multilingual tokenizer through cross-lingual vocabulary pruning and frequency-based analysis to reduce vocabulary size. Next, a teacher model is cloned and its embedding layer is reinitialized for compatibility. Finally, offline distillation is performed using precomputed target vectors. This method reduces model parameters by 33% and achieves full training in under four hours on a single GPU at a cost of merely \$5–20. On the STSb-TR benchmark, it attains a Pearson correlation of 77.55%, and scores 63.9% on average across the TR-MTEB suite, outperforming even larger teacher models.

Abstract

Sentence embeddings are a foundational component for semantic search, clustering, classification, and retrieval-augmented generation. This paper presents embeddingmagibu-200m, a Turkish-focused sentence embedding model that produces 768-dimensional L2-normalized vectors and supports an 8,192-token context window, far exceeding the 512-token limit of earlier BERT-based Turkish encoders. Instead of full pretraining, an efficient three-stage adaptation pipeline is introduced: (1) construct a Turkish-optimized multilingual tokenizer with a 131,072 vocabulary by pruning redundant tokens from the teacher's vocabulary and incorporating multilingual tokens via frequency analysis on a 40-language corpus, (2) clone a teacher embedding model while preserving transformer backbone weights and initializing a compatible embedding table for the new vocabulary via mean-composition token mapping, and (3) perform offline embedding distillation from precomputed teacher vectors using a cosine similarity objective over a balanced 40-language Wikipedia corpus. The resulting student model contains approximately 200M parameters and trains in roughly four hours on a single GPU by avoiding online teacher inference during training, at a total cost of $5-$20. Empirically, Pearson/Spearman correlations of 77.55%/77.45% are obtained on STSbTR, surpassing the 300M-parameter teacher model (73.84%/72.92%). On TR-MTEB (26 tasks), a mean score of 63.9% is achieved (7th out of 26 models), providing a competitive cost-quality trade-off with 33% fewer parameters than the teacher. To facilitate reproducibility and downstream use, all artifacts are released including model weights, tokenizer files, precomputed embedding datasets, and open-source cloning and distillation tooling.

Problem

• Turkish sentence embeddings
• multilingual embedding adaptation
• tokenizer optimization
• embedding distillation
• low-resource language

Innovation

• cross-lingual tokenizer surgery
• offline embedding distillation
• sentence embedding adaptation
• multilingual model transfer
• efficient model distillation