Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale

2026-02-04

arXiv.org

AI Summary

This work addresses the lack of a unified understanding of diverse mechanistic phenomena in Transformer language models—such as induction heads, functional vectors, and the Hydra effect—and their interplay with data, architecture, and optimization. We propose that these phenomena collectively arise from hierarchical latent structures inherent in the data-generating process, shaped by decorrelating gradient dynamics among model components and directional concavity in representational geometry. Through theoretical analysis, toy models, large-scale synthetic data experiments, and comparisons with real language model behaviors, we unify these seemingly disparate phenomena under a common framework rooted in data hierarchy and optimization geometry. Our findings reveal shared mechanisms across models and scales, with extensive empirical evidence strongly supporting the proposed framework.

Abstract

Contemporary studies have uncovered many puzzling phenomena in the neural information processing of Transformer-based language models. Building a robust, unified understanding of these phenomena requires disassembling a model within the scope of its training. While the intractable scale of pretraining corpora limits a bottom-up investigation in this direction, simplistic assumptions of the data generation process limit the expressivity and fail to explain complex patterns. In this work, we use probabilistic context-free grammars (PCFGs) to generate synthetic corpora that are faithful and computationally efficient proxies for web-scale text corpora. We investigate the emergence of three mechanistic phenomena: induction heads, function vectors, and the Hydra effect, under our designed data generation process, as well as in the checkpoints of real-world language models. Our findings suggest that hierarchical structures in the data generation process serve as the X-factor in explaining the emergence of these phenomena. We provide the theoretical underpinnings of the role played by hierarchy in the training dynamics of language models. In a nutshell, our work is the first of its kind to provide a unified explanation behind the emergence of seemingly unrelated mechanistic phenomena in LLMs, augmented with efficient synthetic tooling for future interpretability research.

Problem

• mechanistic interpretability
• hierarchical latent structures
• data generation process
• Transformer language models
• induction heads

Innovation

• hierarchical latent structures
• mechanistic interpretability
• induction heads
• decorrelated gradients
• representation geometry