Uncovering Symmetry Transfer in Large Language Models via Layer-Peeled Optimization

2026-05-12

AI Summary

This work investigates whether large language models (LLMs), trained under standard pretraining objectives, can spontaneously learn geometric structures aligned with the symmetry of the target token distribution. By constructing an analytically tractable layer-peeled optimization model that treats the output projection matrix and the final-layer contextual embeddings as variables, the authors rigorously demonstrate—using tools from group theory, circulant matrices, and equiangular tight frame (ETF) theory—that cyclic symmetry induces circulant logits and Gram matrices, while permutation symmetry leads to simplex ETF structures. This study is the first to reveal, through the lens of group actions, the precise mechanism by which symmetry propagates to globally optimal solutions without explicit regularization. The theoretical predictions are empirically validated on open-source LLMs.

Abstract

Large language models (LLMs) are pretrained by minimizing the cross-entropy loss for next-token prediction. In this paper, we study whether this optimization strategy can induce geometric structure in the learned model weights and context embeddings. We approach this problem by analyzing a constrained layer-peeled optimization program, which serves as a mathematically tractable surrogate for LLMs by treating the output projection matrix and last-layer context embeddings as optimization variables. Our analysis of this nonconvex optimization program demonstrates that symmetries in the target next-token distributions are transferred to the global minimizers of the layer-peeled model in a precise group-theoretic sense. Specifically, we prove that when the target tokens exhibit a cyclic-shift symmetry (such as the seven days of the week or the twelve months of the year), the optimal logit matrix is exactly circulant, and the Gram matrices of both the output projections and the context embeddings form circulant geometries as well. Next, for exchangeable target distributions invariant under the symmetric group and, more generally, under two-transitive group actions, we show that the global optimal output projection matrix forms a simplex equiangular tight frame, while the optimal logit matrix and context embeddings inherit the permutation symmetries present in the input data. A key technical step is to reduce the constrained nonconvex factorized problem to an explicit logit-level convex characterization for cyclic symmetry and to a symmetry-based lower bound for permutation symmetry, together with a sharp characterization of the optimal factorization. Finally, we empirically demonstrate that open-source LLMs naturally exhibit symmetries consistent with our theoretical predictions, despite being trained without any explicit regularization promoting such geometric structure.

Problem

• symmetry transfer
• large language models
• geometric structure
• next-token prediction
• group-theoretic symmetry

Innovation

• symmetry transfer
• layer-peeled optimization
• circulant structure
• equiangular tight frame
• group-theoretic analysis