The Structure of Relation Decoding Linear Operators in Large Language Models

2025-10-30

AI Summary

This work investigates the structural properties and generalization mechanisms of linear operators responsible for decoding relational facts in large language models (LLMs). We find that such operators do not precisely encode individual relations but instead extract coarse-grained, cross-relation semantic attributes—exhibiting an attribute-centric organizational structure. Their parameters are highly redundant and can be compressed over 90% via third-order tensor networks with negligible accuracy loss. To rigorously assess generalization, we propose a cross-relation evaluation protocol, demonstrating that operator transfer stems from semantic attribute reuse rather than relation-specific fitting. Methodologically, we integrate linear operator analysis, tensor network modeling, and cross-validation to systematically uncover structural commonalities and compression potential across multi-relation decoders. Our findings provide novel theoretical insight into how LLMs represent relational knowledge and offer a practical pathway toward lightweight, interpretable model deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Relational Probabilistic Models
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This paper investigates the structure of linear operators introduced in Hernandez et al. [2023] that decode specific relational facts in transformer language models. We extend their single-relation findings to a collection of relations and systematically chart their organization. We show that such collections of relation decoders can be highly compressed by simple order-3 tensor networks without significant loss in decoding accuracy. To explain this surprising redundancy, we develop a cross-evaluation protocol, in which we apply each linear decoder operator to the subjects of every other relation. Our results reveal that these linear maps do not encode distinct relations, but extract recurring, coarse-grained semantic properties (e.g., country of capital city and country of food are both in the country-of-X property). This property-centric structure clarifies both the operators' compressibility and highlights why they generalize only to new relations that are semantically close. Our findings thus interpret linear relational decoding in transformer language models as primarily property-based, rather than relation-specific.

Problem

• Investigating structure of linear operators decoding relational facts
• Explaining compressibility via recurring semantic property extraction
• Interpreting relational decoding as property-based rather than relation-specific

Innovation

• Compress relation decoders via tensor networks
• Cross-evaluation reveals semantic property extraction
• Linear operators decode properties not distinct relations