Code-Switching Reveals Language Anchoring in Multilingual LLMs

2026-06-17

AI Summary

This work addresses the significant performance degradation of multilingual large language models on code-mixed inputs, a phenomenon whose underlying mechanisms remain poorly understood. The study introduces the concept of “anchoring bias” and, through a syntactically constrained code-mixing setup, reveals an asymmetric geometric misalignment between representations of the source and target languages. To mitigate this issue, the authors propose CANVAS, a reasoning-time intervention that leverages internal anchoring signals during the prefill phase to softly steer the target-language hidden states toward source-language anchor points. Experimental results demonstrate that CANVAS consistently improves F1 scores across diverse multilingual models and code-mixing scenarios in question-answering tasks, effectively alleviating performance degradation without requiring model retraining.

Abstract

Multilingual Large Language Models (MLLMs) are increasingly expected to handle Code-Switched (CS) inputs, yet mixing languages frequently degrades performance relative to source- or target-language monolingual counterparts. To understand this degradation, we use grammar-forced CS as a controlled diagnostic setting for locating CS representations relative to their source and target counterparts. We introduce Anchor Bias, a geometric measure that quantifies language anchoring, whether a CS hidden state aligns closer to its source or target language counterpart. Across diverse MLLMs, Anchor Bias reveals a consistent grammar-frame effect: source-framed CS stays source-anchored, whereas target-framed CS shifts target-ward and shows larger Question Answering (QA) degradation. Motivated by this representational pattern, we propose CANVAS (Contextual Anchor-based Neural Vector Alignment Steering), an inference-time intervention that extracts a source-side canvas from the input and softly steers target-language hidden states toward the source anchor during prefill. CANVAS consistently recovers QA F1 across MLLMs and CS conditions, showing that internal anchoring signals provide an actionable target for mitigating CS inference failures.

Problem

• Code-Switching
• Multilingual LLMs
• Language Anchoring
• Anchor Bias
• Question Answering

Innovation

• Anchor Bias
• Code-Switching
• CANVAS
• Language Anchoring
• Multilingual LLMs