Sparsity, Superposition, and Forgetting: A Mechanistic Study of Representation Retention in Continual Learning

2026-06-18

AI Summary

This work addresses the challenge of disentangling model forgetting mechanisms in continual learning under real-world data, where such dynamics are difficult to isolate. To overcome this, the authors construct a controllable synthetic data environment using a generator–separator pipeline that explicitly defines latent feature structures, enabling precise manipulation of task sparsity and overlap. They introduce quantifiable metrics—representation strength and hyperposition—and integrate dynamic system modeling (SINDy) with task-level effective rank analysis to establish an interpretable and verifiable framework for mechanism analysis. Their findings reveal that hyperposition generally increases over time but transiently drops during task switches; high sparsity enhances hyperposition without necessarily causing forgetting; and under sparse conditions, tasks exhibit higher effective rank, indicating more efficient utilization of representational capacity. These results challenge the prevailing notion that hyperposition inevitably leads to catastrophic forgetting.

Abstract

Continual learning (CL) systems often forget previously acquired knowledge, yet the mechanisms driving forgetting remain hard to isolate in practice because real datasets entangle many factors. We present a controlled, toy-world framework that makes these mechanisms observable and testable. Using a synthetic generator-separator pipeline, we define ground-truth latent features, build tasks with tunable sparsity and overlap, and introduce measurable quantities for representation strength and superposition (directional overlap among features). We then study retention dynamics-the temporal change of representation strength by fitting sparse dynamical relations (via SINDy) between retention, superposition, and exposure history. A complementary task-level analysis based on effective rank characterizes how representational capacity is allocated across tasks. Our controlled experiments yield three takeaways. (1) Superposition tends to increase over time with transient dips at task boundaries, suggesting boundary-specific interference rather than steady drift. (2) Higher feature sparsity induces more superposition yet does not inevitably cause forgetting; when representations remain strong, forgetting can be reduced despite overlap. (3) Task-level effective rank grows with sparsity, indicating broader capacity usage under sparse regimes. Together, these results nuance the common intuition that more superposition leads to more forgetting by showing that overlap interacts with representation strength and capacity allocation. Our toy analysis provides falsifiable hypotheses and diagnostic tools for CL.

Problem

• continual learning
• forgetting
• superposition
• sparsity
• representation retention

Innovation

• superposition
• sparsity
• continual learning
• representation retention
• effective rank