Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models

2025-11-10

AI Summary

This work investigates the intrinsic mechanisms by which multi-head self-attention (MHA) in multilingual large language models (LLMs) enables cross-lingual transfer. To this end, we propose a gradient-based sensitivity metric to quantify the importance of individual attention heads per language—yielding the first empirical evidence of coexisting language-specific and language-general attention heads. Building on this insight, we design a lightweight soft attention head masking mechanism with only 20 trainable parameters, enabling fine-grained language-aware attention redistribution and effective suppression of non-target language generation. We validate our approach on Aya-23-8B, Llama-3.2-3B, and Mistral-7B-v0.1, demonstrating significant improvements in XQuAD accuracy while simultaneously enhancing cross-lingual understanding capabilities and model interpretability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large language models (LLMs) increasingly support multilingual understanding and generation. Meanwhile, efforts to interpret their internal mechanisms have emerged, offering insights to enhance multilingual performance. While multi-head self-attention (MHA) has proven critical in many areas, its role in multilingual capabilities remains underexplored. In this work, we study the contribution of MHA in supporting multilingual processing in LLMs. We propose Language Attention Head Importance Scores (LAHIS), an effective and efficient method that identifies attention head importance for multilingual capabilities via a single forward and backward pass through the LLM. Applying LAHIS to Aya-23-8B, Llama-3.2-3B, and Mistral-7B-v0.1, we reveal the existence of both language-specific and language-general heads. Language-specific heads enable cross-lingual attention transfer to guide the model toward target language contexts and mitigate off-target language generation issue, contributing to addressing challenges in multilingual LLMs. We also introduce a lightweight adaptation that learns a soft head mask to modulate attention outputs over language heads, requiring only 20 tunable parameters to improve XQuAD accuracy. Overall, our work enhances both the interpretability and multilingual capabilities of LLMs from the perspective of MHA.

Problem

• Investigating attention head roles in multilingual large language models
• Identifying language-specific and general heads for cross-lingual transfer
• Developing lightweight adaptation to improve multilingual question answering

Innovation

• LAHIS method identifies multilingual attention head importance
• Language-specific heads enable cross-lingual attention transfer
• Lightweight adaptation learns soft mask with few parameters