A Gauge Theory of Superposition: Toward a Sheaf-Theoretic Atlas of Neural Representations

2026-02-28

AI Summary

This work addresses the lack of global interpretability in hyperpositional representations within large language models by proposing a theoretical framework grounded in discrete gauge theory and sheaf theory. Hyperpositions are modeled as sheafed contextual complexes composed of local semantic charts, each equipped with an information-geometric metric to characterize feature interactions. The study formally introduces three measurable obstructions—local congestion, proxy shear, and nontrivial holonomy—and establishes their theoretical connections to interpretability. Leveraging tools from discrete gauge theory, Fisher/Gauss–Newton metrics, and spanning-tree gauge fixing, the authors validate on Llama 3.2 3B Instruct that holonomy is gauge-invariant, derive a lower bound on transfer mismatch induced by shear, and provide certified bounds and estimation stability for congestion and interference effects.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

We develop a discrete gauge-theoretic framework for superposition in large language models (LLMs) that replaces the single-global-dictionary premise with a sheaf-theoretic atlas of local semantic charts. Contexts are clustered into a stratified context complex; each chart carries a local feature space and a local information-geometric metric (Fisher/Gauss--Newton) identifying predictively consequential feature interactions. This yields a Fisher-weighted interference energy and three measurable obstructions to global interpretability: (O1) local jamming (active load exceeds Fisher bandwidth), (O2) proxy shearing (mismatch between geometric transport and a fixed correspondence proxy), and (O3) nontrivial holonomy (path-dependent transport around loops). We prove and instantiate four results on a frozen open LLM (Llama~3.2~3B Instruct) using WikiText-103, a C4-derived English web-text subset, and \texttt{the-stack-smol}. (A) After constructive gauge fixing on a spanning tree, each chord residual equals the holonomy of its fundamental cycle, making holonomy computable and gauge-invariant. (B) Shearing lower-bounds a data-dependent transfer mismatch energy, turning $D_{\mathrm{shear}}$ into an unavoidable failure bound. (C) We obtain non-vacuous certified jamming/interference bounds with high coverage and zero violations across seeds/hyperparameters. (D) Bootstrap and sample-size experiments show stable estimation of $D_{\mathrm{shear}}$ and $D_{\mathrm{hol}}$, with improved concentration on well-conditioned subsystems.

Problem

• superposition
• global interpretability
• semantic representation
• large language models
• sheaf theory

Innovation

• gauge theory
• sheaf theory
• information geometry
• neural representation
• holonomy