Feature Resemblance: On the Theoretical Understanding of Analogical Reasoning in Transformers

2026-03-05

AI Summary

Existing evaluation methods conflate multiple reasoning types, making it difficult to isolate and study analogical reasoning mechanisms. This work addresses this gap through theoretical analysis and large-scale experiments (up to 1.5B parameters), providing the first evidence that Transformers perform analogical reasoning via representational alignment. The study further reveals the critical influence of training order and data structure on reasoning capabilities. Key contributions include the proposal of a representational alignment mechanism, the demonstration of the necessity of curriculum learning for effective analogical reasoning, and the unification of two-hop reasoning as a form of analogical reasoning—thereby elucidating the underlying principles of multi-hop reasoning in Transformer models.

Technology Category

• Cognitive Modeling & Cognitive Systems: Analogy
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Computer Vision: Visual Reasoning & Symbolic Representations

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Understanding reasoning in large language models is complicated by evaluations that conflate multiple reasoning types. We isolate analogical reasoning (inferring shared properties between entities based on known similarities) and analyze its emergence in transformers. We theoretically prove three key results: (1) Joint training on similarity and attribution premises enables analogical reasoning through aligned representations; (2) Sequential training succeeds only when similarity structure is learned before specific attributes, revealing a necessary curriculum; (3) Two-hop reasoning ($a \to b, b \to c \implies a \to c$) reduces to analogical reasoning with identity bridges ($b = b$), which must appear explicitly in training data. These results reveal a unified mechanism: transformers encode entities with similar properties into similar representations, enabling property transfer through feature alignment. Experiments with architectures up to 1.5B parameters validate our theory and demonstrate how representational geometry shapes inductive reasoning capabilities.

Problem

• analogical reasoning
• transformers
• feature alignment
• inductive reasoning
• representational geometry

Innovation

• analogical reasoning
• feature alignment
• representational geometry
• curriculum learning
• two-hop reasoning