Differential syntactic and semantic encoding in LLMs

2026-01-08

arXiv.org

AI Summary

This study investigates the encoding mechanisms and distinctions between syntactic and semantic information in the internal representations of large language models. By computing centroids—mean hidden representations—of sentences sharing either syntactic structure or semantic content, and combining vector subtraction with cross-layer similarity analyses, the work reveals for the first time a partially disentangled linear encoding pattern for syntax and semantics within the model. The findings demonstrate that syntactic and semantic centroids significantly influence the vector similarity of corresponding sentences, and that their encoding trajectories diverge markedly across model layers. This layer-wise differentiation suggests that syntactic and semantic information is organized in distinct ways within deep contextual representations.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Symbolic Representations

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We study how syntactic and semantic information is encoded in inner layer representations of Large Language Models (LLMs), focusing on the very large DeepSeek-V3. We find that, by averaging hidden-representation vectors of sentences sharing syntactic structure or meaning, we obtain vectors that capture a significant proportion of the syntactic and semantic information contained in the representations. In particular, subtracting these syntactic and semantic ``centroids''from sentence vectors strongly affects their similarity with syntactically and semantically matched sentences, respectively, suggesting that syntax and semantics are, at least partially, linearly encoded. We also find that the cross-layer encoding profiles of syntax and semantics are different, and that the two signals can to some extent be decoupled, suggesting differential encoding of these two types of linguistic information in LLM representations.

Problem

• syntactic encoding
• semantic encoding
• large language models
• representation analysis
• differential encoding

Innovation

• differential encoding
• syntactic representation
• semantic representation
• centroid analysis
• linear encoding