Invariant Features in Language Models: Geometric Characterization and Model Attribution

2026-05-07

AI Summary

Although language models exhibit strong robustness to paraphrasing, the underlying mechanisms preserving semantic invariance remain unclear. This work proposes a local geometric framework to characterize, for the first time, the structural properties of semantically equivalent inputs in latent space, revealing that semantic and non-semantic perturbations correspond to distinct subspace directions. To disentangle these variations, we introduce a contrastive subspace discovery method that effectively separates semantic from non-semantic components and leverages the extracted invariant representations for zero-shot model attribution. Experiments demonstrate that this invariant structure is concentrated at specific network depths, with semantic shifts predominantly lying outside the perturbation subspace. Moreover, the invariant components exert a causal influence on model outputs, significantly improving attribution accuracy.

Abstract

Language models exhibit strong robustness to paraphrasing, suggesting that semantic information may be encoded through stable internal representations, yet the structure and origin of such invariance remain unclear. We propose a local geometric framework in which semantically equivalent inputs occupy structured regions in latent space, with paraphrastic variation along nuisance directions and semantic identity preserved in invariant subspaces. Building on this view, we make three contributions: (1) a geometric characterization of invariant latent features, (2) a contrastive subspace discovery method that separates semantic-changing from semantic-preserving variation, and (3) an application of invariant representations to zero-shot model attribution. Across models and layers, empirical results support these contributions. Invariant structure emerges in specific depth regions, semantic displacement lies largely outside the nuisance subspace, and representation-level interventions indicate a causal role of invariant components in model outputs. Invariant representations also capture model-specific geometric patterns, enabling accurate attribution. These findings suggest that semantic invariance can be viewed as a local geometric property of latent representations, offering a principled perspective on how language models organize meaning.

Problem

• invariant features
• semantic invariance
• latent representations
• language models
• geometric characterization

Innovation

• invariant representations
• geometric characterization
• contrastive subspace discovery
• nuisance directions
• zero-shot model attribution