Abstract representational geometry supports inference in large language models

2026-06-22

AI Summary

This study investigates whether large language models (LLMs) achieve reasoning through hippocampus-like abstract representational mechanisms rather than merely exploiting task-specific statistical regularities. We introduce a textual variant of the contextual reversal learning paradigm and employ representational geometry analysis, intervention experiments—including task-sequence modeling and high-level geometric regularization—and cross-layer neural tracking to systematically compare humans and LLMs at both behavioral and representational levels. We report the first evidence that successful reasoning in LLMs is associated with low-dimensional, near-orthogonal manifolds in high-level activations, reflecting an abstract contextual geometry that is hierarchically organized and causally contributes to reasoning. Moreover, geometric disentanglement can induce reasoning capabilities, and high-level geometric regularization substantially enhances generalization in reasoning tasks.

Abstract

A defining feature of human intelligence is the ability to adapt to changing environments by inferring latent task structure from sparse observations. Neuroscientific research indicates that this capability relies on the hippocampus constructing abstract representations, expressed as low-dimensional, approximately orthogonal manifolds in neural state space. However, the internal mechanisms of large language models (LLMs) remain largely opaque, making it unclear whether they form comparable abstract representations or instead rely on task-specific statistical regularities when performing comparable reasoning tasks. Here we adapt a contextual reversal-learning paradigm to a text-based setting and compare humans and LLMs at both the Behavioural and representational levels. We report that although LLMs exhibit generalizable reasoning less frequently than humans, when such inference occurs, their internal states exhibit abstract geometric structures that resemble those reported in the hippocampus. Notably, this representational geometry is not uniformly distributed but is organized hierarchically across model depth: whereas lower layers show early, stable encoding of stimulus identity, higher layers form a hippocampal-like functional band enriched for abstract context geometry associated with inference. Furthermore, complementary intervention experiments mechanistically implicate geometry in reasoning: task-sequence language modelling induces geometric disentanglement, whereas geometric regularization of higher layers increases the emergence of generalizable inference. Together, these findings establish abstract representational geometry as a mechanistic principle supporting inference in large language models.

Problem

• abstract representational geometry
• large language models
• inference
• hippocampus
• reasoning

Innovation

• abstract representational geometry
• large language models
• hippocampal-like representations
• hierarchical organization
• geometric regularization