Large Language Models Are Cross-Lingual Knowledge-Free Reasoners

2024-06-24

arXiv.org

AI Summary

This work investigates the intrinsic mechanisms underlying cross-lingual reasoning in large language models (LLMs), proposing a decomposition of reasoning into two orthogonal components: knowledge retrieval and knowledge-agnostic reasoning. To isolate these, we construct a novel knowledge-agnostic reasoning dataset and adapt commonsense reasoning benchmarks to minimize knowledge dependency. We further employ hidden-state similarity analysis and quantification of feed-forward network (FFN) neuron activation overlap across languages. Our analysis reveals, for the first time, that knowledge-agnostic reasoning achieves >95% average transfer accuracy across 12+ language pairs—demonstrating near-perfect cross-lingual transferability—and critically relies on shared, cross-lingual neurons. In contrast, knowledge retrieval exhibits substantially lower transfer performance due to language-specific knowledge encoding. This decoupled framework establishes a new paradigm for understanding the nature of multilingual reasoning in LLMs and provides a theoretical foundation for efficient cross-lingual reasoning modeling.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models have demonstrated impressive reasoning capabilities across multiple languages. However, the relationship between capabilities in different languages is less explored. In this work, we decompose the process of reasoning tasks into two separated components: knowledge retrieval and knowledge-free reasoning, and analyze the relationship between cross-lingual transferability and these two components. With adapted commonsense reasoning datasets and constructed knowledge-free reasoning datasets, we show that the knowledge-free reasoning capability can be nearly perfectly transferred across various source-target language directions despite the secondary impact of resource in some specific target languages, while cross-lingual knowledge retrieval significantly hinders the transfer. Moreover, by analyzing the hidden states and feed-forward network neuron activation during the reasoning, we show that higher similarity of hidden representations and larger overlap of activated neurons could explain the better cross-lingual transferability of knowledge-free reasoning than knowledge retrieval. Thus, we hypothesize that knowledge-free reasoning shares similar neurons in different languages for reasoning, while knowledge is stored separately in different languages. Our code and data is available at: https://github.com/NJUNLP/Knowledge-Free-Reasoning.

Problem

• Analyze cross-lingual transferability in reasoning tasks.
• Separate reasoning into knowledge retrieval and knowledge-free components.
• Investigate neuron activation patterns in cross-lingual reasoning.

Innovation

• Decompose reasoning into knowledge retrieval and knowledge-free components
• Analyze cross-lingual transferability via hidden states and neuron activation
• Show knowledge-free reasoning transfers better across languages