The Lattice Representation Hypothesis of Large Language Models

2026-03-01

AI Summary

This work addresses the challenge of realizing symbolic conceptual hierarchies and logical reasoning within the continuous embedding space of large language models. It proposes the "lattice representation hypothesis," introducing formal concept analysis into the study of language model representations for the first time. By constructing concept lattices through intersections of half-spaces induced by linear attribute directions, the approach unifies formal concept analysis with the linear representation hypothesis. This framework enables symbolic reasoning via geometric meet (intersection) and join (union) operations. Experiments on WordNet sub-hierarchies demonstrate that the embedding space indeed encodes interpretable concept lattice structures, thereby establishing a principled bridge between continuous geometric representations and symbolic abstraction.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

We propose the Lattice Representation Hypothesis of large language models: a symbolic backbone that grounds conceptual hierarchies and logical operations in embedding geometry. Our framework unifies the Linear Representation Hypothesis with Formal Concept Analysis (FCA), showing that linear attribute directions with separating thresholds induce a concept lattice via half-space intersections. This geometry enables symbolic reasoning through geometric meet (intersection) and join (union) operations, and admits a canonical form when attribute directions are linearly independent. Experiments on WordNet sub-hierarchies provide empirical evidence that LLM embeddings encode concept lattices and their logical structure, revealing a principled bridge between continuous geometry and symbolic abstraction.

Problem

• Lattice Representation
• Large Language Models
• Concept Hierarchies
• Symbolic Reasoning
• Embedding Geometry

Innovation

• Lattice Representation
• Formal Concept Analysis
• Symbolic Reasoning
• Embedding Geometry
• Concept Hierarchy