When Language Representations Interact: Separability and Cross-Lingual Effects in LLMs

2026-06-12

AI Summary

This study addresses the limited interpretability of internal representations in multilingual large language models and the complex inter-language dependencies that lead to unstable cross-lingual behavior. For the first time, the authors extend a causal geometric interpretability framework to multilingual settings, systematically analyzing representations across 28 bilingual pairs through covariance-adjusted inner products, linear separability tests, and geometric structure modeling. Their analysis reveals that, under covariance-adjusted inner products, linguistic concepts exhibit stable linear representations. Furthermore, languages within the same family form simplex-like geometric structures, uncovering hierarchical and predictable patterns of cross-lingual dependence. These findings provide a principled geometric understanding of how multilingual models encode and relate linguistic knowledge across diverse languages.

Abstract

Large language models exhibit strong multilingual capabilities, however, their internal representations are difficult to interpret. Understanding these interactions is important for ensuring reliable behavior in multilingual systems. Recent work has shown that causal-geometric structure can explain how certain concepts are encoded as approximately linear and separable directions, but whether this framework extends to multilingual models, where language identity is correlated and hierarchical, is underexplored. We apply causal-geometric analysis to multilingual LLMs, studying 28 bilingual contrasts across three models, allowing us to analyze when languages behave as approximately independent factors and when structured dependencies persist. We find evidence that language concepts admit stable linear representations that are largely separable under a covariance-adjusted (causal) inner product, with structured deviations reflecting linguistic similarity. Moreover, languages within the same family (such as Germanic or Romance) exhibit a simplex-like geometric structure, suggesting hierarchical organization. These results extend causal-geometric interpretability to multilingual settings and provide insight into how separability and similarity may exist in multilingual LLM representations, motivating interpretability analyses that diagnose when and how structured dependencies between concepts can be anticipated. This has implications for trustworthy deployment, as residual structure between languages may lead to unintended cross-lingual effects when models are monitored or intervened upon.

Problem

• multilingual LLMs
• language representations
• separability
• cross-lingual effects
• causal-geometric structure

Innovation

• causal-geometric analysis
• multilingual LLMs
• language separability
• structured dependencies
• simplex-like geometry