The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers

2026-08-25

AI Summary

本文研究了Transformer模型中不同词性在各层的几何变化，通过内在维度和邻域结构的变化来解释这种现象，并比较了编码器和解码器的不同演变方式。

Abstract

Transformer representations describe trajectories through high-dimensional vector spaces, which are shaped dynamically as tokens incorporate relational context across layers. Such data tend to concentrate on lower-dimensional sub-manifolds, a form of compression quantified by the Intrinsic Dimensionality (ID), the minimum number of independent variables needed to represent them without significant information loss. In this work, we ask whether the grammatical role of tokens, as marked by their part-of-speech (PoS) tag, shapes the local geometry of this manifold. To this end: (1) We investigate the layer-wise evolution of ID, finding that closed-class items expand earlier and collapse sooner than open-class ones; (2) We show its expansion and contraction to be explained by changes in the neighborhood structure, and hence in the relations between words within a sentence; (3) We compare encoders (ModernBERT, bigbird-roberta-large) and decoders (gemma-2-2B, Llama-3.2-3B), finding that the two families evolve differently across layers, consistently with how each integrates context;(4) We show that geometric features alone recover a token's grammatical role, and use them to interpret how the semantic content of each PoS evolves across layers in a downstream classification task.

Problem

• Intrinsic Dimensionality
• neighborhood structure
• part-of-speech (PoS) tag
• Transformer layers

Innovation

• Intrinsic Dimensionality
• neighborhood structure
• part-of-speech