The Geometry of Low-Resource Language Representations

2026-08-24

AI Summary

研究通过几何特性分析低资源语言在大模型中的表示退化问题，并提出正则化方法以改善继续预训练过程中的表现。

Abstract

The performance gap between low- and high-resource languages in LLMs is widely known, but it remains unclear which internal model factors drive these disparities. In this paper, we characterise this gap through the lens of representational geometry. Comparing the geometric properties of hidden representations across 30 languages reveals that LLM geometry is systematically related to language data availability. The most consistent effect is in final layers, where low-resource languages exhibit representational degeneration. To counter this, we investigate the effectiveness of regularisation terms to penalise degeneration during continued pretraining (CPT). Experiments monolingually adapting 9 base LLMs to 10 African languages show that geometric regularisation successfully reduces representational degeneration during CPT. For larger models, cosine similarity-based regularisation marginally improves performance over vanilla CPT, with more consistent gains on the most challenging tasks. We establish that the representational geometry of low- and high-resource languages in LLMs is measurably distinct, and that targeted geometric intervention is a viable strategy for improving CPT for low-resource languages.

Problem

• low-resource languages
• high-resource languages
• performance gap
• representational geometry
• continued pretraining

Innovation

• representational geometry
• low-resource languages
• regularisation terms
• continued pretraining (CPT)
• cosine similarity