Cell-Based Representation of Relational Binding in Language Models

2026-04-20

AI Summary

This study investigates how large language models (LLMs) bind entities, relations, and attributes at the discourse level. To this end, the authors propose the Cellular Binding Representation (CBR) model, which encodes entity–relation index pairs within low-dimensional linear subspaces—termed “cells”—and retrieves corresponding attributes during inference. The work provides the first empirical evidence of a grid-like geometric structure underlying CBR in LLMs, demonstrating its capacity for cross-context generalization. By applying partial least squares regression to decode index information from activations, and combining activation patching with vector arithmetic analyses, the study establishes causal evidence for CBR’s role in relational reasoning. Experiments reveal that this representational structure is prevalent across diverse domains and model architectures; targeted manipulation of the subspace systematically controls relation predictions, while perturbations significantly degrade performance.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Understanding a discourse requires tracking entities and the relations that hold between them. While Large Language Models (LLMs) perform well on relational reasoning, the mechanism by which they bind entities, relations, and attributes remains unclear. We study discourse-level relational binding and show that LLMs encode it via a Cell-based Binding Representation (CBR): a low-dimensional linear subspace in which each ``cell'' corresponds to an entity--relation index pair, and bound attributes are retrieved from the corresponding cell during inference. Using controlled multi-sentence data annotated with entity and relation indices, we identify the CBR subspace by decoding these indices from attribute-token activations with Partial Least Squares regression. Across domains and two model families, the indices are linearly decodable and form a grid-like geometry in the projected space. We further find that context-specific CBR representations are related by translation vectors in activation space, enabling cross-context transfer. Finally, activation patching shows that manipulating this subspace systematically changes relational predictions and that perturbing it disrupts performance, providing causal evidence that LLMs rely on CBR for relational binding.

Problem

• relational binding
• large language models
• entity tracking
• discourse understanding
• representation mechanism

Innovation

• Cell-based Binding Representation
• relational binding
• activation subspace
• linear decodability
• activation patching