Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations

2026-08-26

AI Summary

研究探讨了双语预训练模型中共享语言表示的问题，通过对比单语和双语模型的隐藏状态差异，揭示了仅对齐嵌入空间可能掩盖模型内部处理机制的不同。

Abstract

When researchers compare multilingual models for probing, interpretability, or cross-lingual transfer, they often align embedding spaces and assume that shared-language representations are comparable. We show that this assumption can be premature for decoder-only models. We pretrain paired 310M-parameter models (one English-only, one bilingual) across eight typologically diverse languages, separately controlling for English exposure, total compute, and document overlap. After aligning on shared English vocabulary, we test held-out words and find that token embeddings look similar after alignment, but the deeper hidden states that the model uses for prediction do not. This gap holds for all eight languages and survives controls for document overlap and alternative alignment methods. This hidden-state mismatch grows through middle transformer layers, suggesting that it arises from contextual processing rather than the input representations where alignment is performed. Embedding alignment can mask real differences in how models internally represent a shared language, which matters for any downstream study that treats aligned models as interchangeable.

Problem

• multilingual models
• embedding alignment
• hidden states
• contextual processing

Innovation

• hidden-state mismatch
• contextual processing
• multilingual models