Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning

2025-05-16

AI Summary

This work investigates the fundamental mechanisms underlying in-context learning (ICL) in large language models, specifically addressing whether ICL arises from data memorization or symbolic algorithmic reasoning. Methodologically, we integrate full-scale Pythia model checkpoints with linear subspace projection analysis of the residual stream to systematically disentangle memorization- and algorithmicity-driven hypotheses. By tracking multi-stage training dynamics and evaluating downstream ICL performance, we demonstrate that ICL capability emerges progressively—consistently improving beyond memorization baselines—and identify a critical subspace evolution trajectory governing its emergence. Our results reveal ICL as an emergent phenomenon jointly driven by training dynamics, model scale, and the geometric structure of residual stream subspaces. This provides a theoretically grounded, interpretable framework for ICL modeling, enabling principled model optimization and rigorous AI safety evaluation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large-scale Transformer language models (LMs) trained solely on next-token prediction with web-scale data can solve a wide range of tasks after seeing just a few examples. The mechanism behind this capability, known as in-context learning (ICL), remains both controversial and poorly understood. Some studies argue that it is merely the result of memorizing vast amounts of data, while others contend that it reflects a fundamental, symbolic algorithmic development in LMs. In this work, we introduce a suite of investigative tasks and a novel method to systematically investigate ICL by leveraging the full Pythia scaling suite, including interim checkpoints that capture progressively larger amount of training data. By carefully exploring ICL performance on downstream tasks and simultaneously conducting a mechanistic analysis of the residual stream's subspace, we demonstrate that ICL extends beyond mere"memorization"of the training corpus, yet does not amount to the implementation of an independent symbolic algorithm. Our results also clarify several aspects of ICL, including the influence of training dynamics, model capabilities, and elements of mechanistic interpretability. Overall, our work advances the understanding of ICL and its implications, offering model developers insights into potential improvements and providing AI security practitioners with a basis for more informed guidelines.

Problem

• Investigates whether in-context learning relies on memorization or symbolic algorithms
• Explores training dynamics and model capabilities in in-context learning
• Analyzes mechanistic interpretability of in-context learning in Transformers

Innovation

• Leveraging Pythia scaling suite for ICL analysis
• Exploring residual stream subspace mechanistically
• Investigating training dynamics and model capabilities