The Curious Case of Analogies: Investigating Analogical Reasoning in Large Language Models

2025-11-25

AI Summary

This study investigates whether large language models (LLMs) can encode high-level relational concepts and achieve structured analogical transfer across contexts. Addressing proportional and narrative analogy tasks, we propose three analytical methods: hidden-state probing, targeted hidden-state patching at critical layers, and structural alignment measurement—systematically characterizing LLMs’ dynamic mechanisms in relational representation, information propagation, and structural alignment. We find that LLMs possess only limited, context-sensitive relational encoding capacity; successful analogical transfer strictly depends on coordinated propagation of attribute and relational information in mid-to-upper transformer layers and strong structural alignment; failure primarily stems from relational information decay or alignment degradation. Our key contributions are twofold: (1) the first demonstration that targeted hidden-state patching can directionally enhance relational transfer, and (2) the establishment of structural alignment as a core intrinsic mechanism underpinning analogical reasoning in LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Analogy
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Analogical reasoning is at the core of human cognition, serving as an important foundation for a variety of intellectual activities. While prior work has shown that LLMs can represent task patterns and surface-level concepts, it remains unclear whether these models can encode high-level relational concepts and apply them to novel situations through structured comparisons. In this work, we explore this fundamental aspect using proportional and story analogies, and identify three key findings. First, LLMs effectively encode the underlying relationships between analogous entities; both attributive and relational information propagate through mid-upper layers in correct cases, whereas reasoning failures reflect missing relational information within these layers. Second, unlike humans, LLMs often struggle not only when relational information is missing, but also when attempting to apply it to new entities. In such cases, strategically patching hidden representations at critical token positions can facilitate information transfer to a certain extent. Lastly, successful analogical reasoning in LLMs is marked by strong structural alignment between analogous situations, whereas failures often reflect degraded or misplaced alignment. Overall, our findings reveal that LLMs exhibit emerging but limited capabilities in encoding and applying high-level relational concepts, highlighting both parallels and gaps with human cognition.

Problem

• Investigating analogical reasoning capabilities in large language models
• Assessing LLMs' encoding of high-level relational concepts for novel situations
• Identifying structural alignment limitations in LLM analogical reasoning

Innovation

• LLMs encode relational information in mid-upper layers
• Strategic patching of hidden representations aids reasoning
• Structural alignment between analogies enables successful reasoning