On Representational Dissociation of Language and Arithmetic in Large Language Models

2025-02-17

AI Summary

This work investigates whether language understanding and arithmetic reasoning are functionally segregated in the internal representational spaces of large language models (LLMs). Using elementary arithmetic as a non-linguistic cognitive proxy, we systematically analyze representational geometry across all transformer layers. Through linear discriminant analysis, inter-cluster separability metrics, and controlled stimuli (e.g., spelled-out arithmetic expressions), we discover— for the first time—that arithmetic expressions and linguistic inputs exhibit fully disjoint representational distributions at every layer, evidencing brain-like functional decoupling. This segregation is highly stable and consistent across diverse LLMs, yet its geometric properties—such as excessive orthogonality—reveal cognitively implausible representational organization. Our findings provide the first empirical evidence supporting functional modularity in LLM cognition and expose a fundamental structural limitation: the absence of integrated, language-thought co-representation in current architectures.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The association between language and (non-linguistic) thinking ability in humans has long been debated, and recently, neuroscientific evidence of brain activity patterns has been considered. Such a scientific context naturally raises an interdisciplinary question -- what about such a language-thought dissociation in large language models (LLMs)? In this paper, as an initial foray, we explore this question by focusing on simple arithmetic skills (e.g., $1+2=$ ?) as a thinking ability and analyzing the geometry of their encoding in LLMs' representation space. Our experiments with linear classifiers and cluster separability tests demonstrate that simple arithmetic equations and general language input are encoded in completely separated regions in LLMs' internal representation space across all the layers, which is also supported with more controlled stimuli (e.g., spelled-out equations). These tentatively suggest that arithmetic reasoning is mapped into a distinct region from general language input, which is in line with the neuroscientific observations of human brain activations, while we also point out their somewhat cognitively implausible geometric properties.

Problem

• Dissociation of language and arithmetic in LLMs
• Encoding geometry in LLMs' representation space
• Comparison with human brain activations

Innovation

• Analyzes arithmetic encoding in LLMs
• Uses linear classifiers for testing
• Explores language-thought dissociation scientifically