Problem
Research questions and friction points this paper is trying to address.
multilingual vision-language encoders
low-resource languages
performance gap
encoder analysis
cross-lingual alignment
Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?
🤖 AI Summary
研究探讨了多语言视觉-语言编码器在低资源语言上的表现不佳问题，通过替换特定层的隐藏状态来提升低资源语言的表现。
📝 Abstract
Recent multilingual vision--language encoders cover hundreds of languages in a single model, yet on two state-of-the-art instances retrieval on low-resource languages (LRL; e.g. Swahili) trails high-resource ones (HRL; e.g. English) by $30^+$\,pp. We ask where in the trained encoder this gap is located. Prior modality-gap and cross-lingual subspace work suggests a linear language direction at the output crowds out alignment-relevant geometry. We falsify this: LEACE drives the linear language classifier from $>99\%$ to near chance and iterated INLP to $37$--$50\%$ while LRL retrieval moves within $\pm 1.5$\,pp and all tier means within $2.2$\,pp, tracking random controls. The linear bias is a \emph{symptom}, not the cause. Instead, the alignment-causal factor lies along the encoder's forward path: the EOS (end-of-sequence) hidden state's per-language trajectory diverges with depth. Substituting the EOS with its parallel English value three blocks before the projector lifts Swahili from $22.1\%$ to $69.1\%$ on one encoder (and reproduces on the other); three controls rule out pooled-position tautology and English specificity. A front-layer trunk that pulls each language's projection toward the parallel-content centroid corroborates the diagnosis at training time, recovering $+9.6$ / $+17.1$\,pp on LRL XM3600 retrieval (1{,}000-image subset), with consistent gains across three further benchmarks while preserving HRL performance.
Innovation
Methods, ideas, or system contributions that make the work stand out.
multilingual vision-language encoders
low-resource languages
end-of-sequence hidden state
alignment-causal factor
front-layer trunk
🔎 Similar Papers
No similar papers found.
