Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering

2025-11-28

AI Summary

To address the fairness issues of large language models (LLMs) and vision-language models (LVLMs) in low-resource languages—namely, weak multilingual reasoning capability and input-output language inconsistency—this paper proposes a training-free inference-time representation engineering method. The core innovation is a novel two-stage vector injection mechanism: (1) injecting precomputed cross-lingual reasoning enhancement vectors at specific network layers to strengthen multilingual logical reasoning, and (2) injecting target-language anchor vectors to ensure output language consistency, jointly optimized with inference-path guidance and latent-space distribution calibration. Extensive experiments across six state-of-the-art models and four reasoning benchmarks demonstrate an average performance gain of 5.48%, with up to 7.54% improvement on low-resource languages and a 3.78% increase in input-output language consistency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) demonstrate strong reasoning capabilities, yet their performance in English significantly outperforms that in low-resource languages, raising fairness concerns in multilingual applications. Existing approaches either rely on costly multilingual training or employ prompting with external translation tools, both of which are resource-intensive and sensitive to translation quality. To address these limitations, we propose a training-free inference-time method to enhance Multilingual Reasoning capabilities via Representation Engineering (MRRE) without using any additional training data or tools. MRRE sequentially injects two precomputed vectors at specific layers during inference processing: cross-lingual reasoning enhancement vectors, which steer non-English reasoning representations toward English space to unlock multilingual reasoning, and target-language output anchoring vectors, which restore the distribution of the target language to preserve input-output language consistency. Comprehensive experiments across six advanced LLMs and LVLMs on four reasoning benchmarks demonstrate that MRRE consistently enhances non-English reasoning by an average gain of 5.48% and up to 7.54% in low-resource languages (Thai and Swahili), while improving input-output language consistency by 3.78%.

Problem

• Enhancing multilingual reasoning in LLMs without training
• Addressing performance gap between English and low-resource languages
• Improving language consistency in multilingual model outputs

Innovation

• Training-free inference-time multilingual reasoning enhancement method
• Injecting cross-lingual reasoning vectors to steer representations
• Using target-language anchoring vectors to maintain language consistency