Intrinsic Task Symmetry Drives Generalization in Algorithmic Tasks

2026-03-02

AI Summary

This study investigates the phenomenon of grokking—abrupt transitions from memorization to generalization in neural networks on algorithmic tasks—and demonstrates that intrinsic task symmetries are a key driver of generalization. By integrating representation analysis, symmetry detection, and training dynamics tracking, the work establishes the first systematic causal link between task symmetries and the geometric structure of representation space, revealing a three-stage evolution involving memorization, symmetry acquisition, and geometric organization. Building on these insights, the authors introduce a symmetry-based diagnostic tool that accurately predicts the onset of generalization across diverse tasks—including algebraic, structural, and relational reasoning—and effectively accelerates generalization while enhancing model robustness.

Technology Category

• Machine Learning: Representation Learning
• Cognitive Modeling & Cognitive Systems: Symbolic Representations
• Computer Vision: Visual Reasoning & Symbolic Representations

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Grokking, the sudden transition from memorization to generalization, is characterized by the emergence of low-dimensional representations, yet the mechanism underlying this organization remains elusive. We propose that intrinsic task symmetries primarily drive grokking and shape the geometry of the model's representation space. We identify a consistent three-stage training dynamic underlying grokking: (i) memorization, (ii) symmetry acquisition, and (iii) geometric organization. We show that generalization emerges during the symmetry acquisition phase, after which representations reorganize into a structured, task-aligned geometry. We validate this symmetry-driven account across diverse algorithmic domains, including algebraic, structural, and relational reasoning tasks. Building on these findings, we introduce a symmetry-based diagnostic that anticipates the onset of generalization and propose strategies to accelerate it. Together, our results establish intrinsic symmetry as the key factor enabling neural networks to move beyond memorization and achieve robust algorithmic reasoning.

Problem

• grokking
• generalization
• intrinsic symmetry
• algorithmic reasoning
• representation geometry

Innovation

• intrinsic symmetry
• grokking
• representation geometry
• algorithmic generalization
• symmetry acquisition