The Structure of Relation Decoding Linear Operators in Large Language Models
This work investigates the structural properties and generalization mechanisms of linear operators responsible for decoding relational facts in large language models (LLMs). We find that such operators do not precisely encode individual relations but instead extract coarse-grained, cross-relation semantic attributes—exhibiting an attribute-centric organizational structure. Their parameters are highly redundant and can be compressed over 90% via third-order tensor networks with negligible accuracy loss. To rigorously assess generalization, we propose a cross-relation evaluation protocol, demonstrating that operator transfer stems from semantic attribute reuse rather than relation-specific fitting. Methodologically, we integrate linear operator analysis, tensor network modeling, and cross-validation to systematically uncover structural commonalities and compression potential across multi-relation decoders. Our findings provide novel theoretical insight into how LLMs represent relational knowledge and offer a practical pathway toward lightweight, interpretable model deployment.