Relational Linear Properties in Language Models: An Empirical Investigation

2026-05-21

AI Summary

This work investigates the existence and variation of relational linearity in language models—specifically, whether a fixed linear mapping exists between subject embeddings and object unembeddings. To address this, the authors propose a KL divergence–based linear probing method that circumvents the coarse approximations of Jacobian matrices used in prior work, substantially improving evaluation accuracy. The approach is systematically applied across multiple layers of language model representations and validated on four datasets. Experimental results reveal that relational linearity is widely present across models, albeit to varying degrees, exhibiting layer-wise patterns aligned with the hierarchical organization of linguistic information and showing strong sensitivity to how relations are expressed.

Abstract

Linear properties are ubiquitous in the representations of language models; however, testing them experimentally remains a challenging task. This work focuses on relational linearity: the hypothesis that, for a fixed relation (e.g., "plays"), the unembedding of an object (e.g., "trumpet") can be predicted from the embedding of its subject (e.g.,"Miles Davis") by a linear map. We present an experimental method to test the formulation of relational linearity by Marconato et al. (2025). Specifically, we introduce a probing method, based on Kullback-Leibler divergence, to evaluate this property and examine its variation across layers and paraphrased relational queries. It is also more efficient than previous work; for example, it avoids the crude Jacobian approximations used in Linear Relational Embeddings by Hernandez et al. (2024). Our findings across four datasets show that relational linearity varies across models, exhibits layer-wise patterns consistent with prior observations about linguistic information in model representations, and is differently affected by changes in how the relation is phrased.

Problem

• relational linearity
• language models
• linear properties
• embedding
• unembedding

Innovation

• relational linearity
• probing method
• Kullback-Leibler divergence
• language model representations
• linear map