Preserving Plasticity in Continual Learning via Dynamical Isometry

2026-06-08

AI Summary

This work addresses the loss of plasticity in deep neural networks during continual learning, a critical limitation caused by non-stationary data distributions that hinders learning on subsequent tasks. For the first time, it explicitly links dynamic isometry to plasticity in continual learning and proposes a synergistic framework comprising an approximately dynamically isometric network architecture, an isometry-promoting regularization technique, and a decoupled optimizer named AdamO to jointly preserve model plasticity. Additionally, it introduces a novel mechanism to reactivate dormant ReLU units and reinterprets the limitations of existing methods through the lens of dynamic isometry. Evaluated across multiple supervised and reinforcement continual learning benchmarks, the proposed approach effectively mitigates plasticity loss and achieves performance on par with or superior to current state-of-the-art methods.

Abstract

Continual training of deep neural networks under non-stationarity often leads to a progressive loss of plasticity, eventually limiting further learning. We relate plasticity to the empirical Neural Tangent Kernel, and identify dynamical isometry (the condition that layer-wise Jacobian singular values remain close to one) as a key mechanism for preserving plasticity in continual learning. We revisit a class of networks that are almost-everywhere isometric while remaining universal Lipschitz function approximators, demonstrating that near-dynamical isometry is compatible with expressive nonlinear representations. For general architectures, we propose an efficient isometry-promoting regularization scheme and identify a novel mechanism by which it can reactivate dormant ReLU units. Building on this, we introduce AdamO, an Adam-style adaptive optimizer that decouples isometry regularization from gradient updates, analogous to AdamW. We further reinterpret prior plasticity-preserving approaches through the lens of dynamical isometry, showing that they target only a partial measure of isometry. Across supervised and reinforcement-learning continual-learning benchmarks designed to induce plasticity loss, our methods consistently match or outperform existing approaches.

Problem

• continual learning
• plasticity loss
• dynamical isometry
• neural networks
• non-stationarity

Innovation

• dynamical isometry
• continual learning
• plasticity preservation
• isometry-promoting regularization
• AdamO