Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation

2026-06-16

AI Summary

This work addresses the challenge in continual learning where training on new tasks often leads to significant performance degradation on previously learned tasks, a phenomenon known as catastrophic forgetting. Existing approaches struggle to identify the directions in output space most susceptible to interference. Building upon the Neural Tangent Kernel (NTK) framework, this study derives a closed-form expression for prediction drift on old tasks in function space, establishing—for the first time—an exact analytical link between forgetting vectors and cross-task kernels. The analysis reveals that forgetting exhibits a low-rank structure and introduces a Kronecker scaling law for the "fragile rank," clarifying its relationship with NTK overlap theory. Under a PEFT-CL setting with frozen backbones and linear heads, the derived expression predicts forgetting with numerical precision, enabling a spectrally regularized method that effectively suppresses interference in output space.

Abstract

Catastrophic forgetting in continual adaptation is usually studied through parameter drift, replay, or distillation, but these views do not identify which output-space directions are vulnerable. We give a function-space account in the NTK regime: new-task training induces old-task prediction drift through the cross-task kernel, yielding a closed-form predictor for the forgetting vector before any new-task gradient step. In frozen-backbone linear-head PEFT-CL, where the model is linear in the trainable parameters, the predictor is exact up to numerical precision; for nonlinear adapters/full fine-tuning, it is a local NTK approximation. The same expression reveals that forgetting concentrates in a small number of old-task NTK eigenmodes and under frozen linear heads gives a Kronecker scaling rule for the vulnerable rank. These results clarify the relation to prior NTK-overlap theory, explain why parameter-space regularizers can miss output-space interference, and motivate a targeted spectral regularizer.

Problem

• Catastrophic Forgetting
• Continual Adaptation
• Function-Space
• NTK
• Output-Space Interference

Innovation

• function-space theory
• catastrophic forgetting
• NTK regime
• low-rank forgetting
• spectral regularizer