Language Models Represent and Transform Concepts with Shared Geometry

2026-07-05

AI Summary

This work investigates the representation of concepts in large language models and their dynamic evolution under contextual influence. Concept representations are modeled as point-cloud manifolds, while contextual effects are formalized as vector fields acting on these manifolds, analyzed through the lens of neural population geometry. The study reveals, for the first time, that language models across different scales and architectures not only share static geometric structures of concepts but also exhibit common dynamic transformation patterns organized along semantic hierarchies. Experiments across six model families demonstrate that concept shifts induced by context are semantically structured, enabling cross-model prediction of such shifts significantly beyond random baselines. These findings uncover a deep geometric consistency in the internal representations of language models, suggesting universal principles underlying their semantic processing.

Abstract

How concepts are represented in neural networks is a fundamental question in machine learning. The dominant view treats concept representations as stationary geometric objects. Yet concepts appear in context, and context transforms them. Drawing from neural population geometry, we formalize concept representations as point-cloud manifolds and contextual transformations as vector fields, and instantiate this framework in large language models. Across six model families of varying scales, we find that context moves each concept differently. The variance in these displacements is semantically organized, correlating with lexical concreteness and density. Importantly, both the concepts being transformed and this variance structure are shared across models: displacement structure transported from one model predicts held-out displacements in others significantly above chance. Together, these findings show that models share a common geometry not only in how concepts are represented, but more importantly in how context transforms them, a structure with richer organization than prior work has recognized.

Problem

• concept representation
• contextual transformation
• neural geometry
• language models
• shared manifold

Innovation

• concept representation
• neural population geometry
• point-cloud manifolds
• contextual transformation
• shared geometry