When Alignment Hurts: Decoupling Representational Spaces in Multilingual Models

2025-08-18

AI Summary

This study investigates the detrimental representational entanglement between high-resource standard languages (e.g., Modern Standard Arabic) and low-resource dialects in multilingual language models. To address this, we propose a subspace-level causal intervention method that jointly leverages geometric and information-theoretic probing: it dynamically decouples the standard-language representation subspace during training via an online variational probing framework, which continuously estimates the standard-language subspace and orthogonally projects it out. Evaluated on fine-tuning for 25 Arabic dialects, our approach yields an average improvement of +2.0 chrF++ in generation quality (up to +4.9), significantly enhancing modeling capability for low-resource variants. This work is the first to integrate subspace-level causal intervention with online variational probing, establishing a scalable and interpretable paradigm for representation disentanglement in low-resource language generation.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Alignment with high-resource standard languages is often assumed to aid the modeling of related low-resource varieties. We challenge this assumption by demonstrating that excessive representational entanglement with a dominant variety, such as Modern Standard Arabic (MSA) in relation to Arabic dialects, can actively hinder generative modeling. We present the first comprehensive causal study of this phenomenon by analyzing and directly intervening in the internal representation geometry of large language models (LLMs). Our key contribution is an online variational probing framework that continuously estimates the subspace of the standard variety during fine-tuning, enabling projection-based decoupling from this space. While our study uses Arabic as a case due to its unusually rich parallel resources across 25 dialects, the broader motivation is methodological: dialectal MT serves as a controlled proxy for generative tasks where comparable multi-variety corpora are unavailable. Across 25 dialects, our intervention improves generation quality by up to +4.9 chrF++ and +2.0 on average compared to standard fine-tuning, despite a measured tradeoff in standard-language performance. These results provide causal evidence that subspace dominance by high-resource varieties can restrict generative capacity for related varieties. More generally, we unify geometric and information-theoretic probing with subspace-level causal interventions, offering practical tools for improving generative modeling in closely related language families and, more broadly, for controlling representational allocation in multilingual and multi-domain LLMs. Code will be released.

Problem

• Excessive alignment with dominant languages hinders low-resource variety modeling
• Decoupling representational spaces improves generative quality for dialects
• Subspace dominance by high-resource languages restricts generative capacity

Innovation

• Online variational probing for subspace estimation
• Projection-based decoupling from dominant variety
• Subspace-level causal interventions in LLMs