Layers Matter: Why Continual Learning Regularization Should Be Layer-Adaptive

2026-08-16

AI Summary

This study addresses the issue in continual learning where per-parameter importance weights fail to accurately capture layer-wise curvature, resulting in uncontrolled catastrophic forgetting. We propose a novel layer-adaptive regularization paradigm, theoretically demonstrating that forgetting is dominated by top-layer eigenvalues and exposing the limitations of diagonal Fisher matrices. Methodologically, leveraging block-diagonal Hessian assumptions and spectral analysis, we design an adaptive mechanism that enforces strong protection for early layers while permitting greater plasticity in deeper layers, subsequently enhancing both EWC and SLCA frameworks. Experimental results demonstrate that this approach significantly outperforms existing baselines in average accuracy and anti-forgetting metrics. Ultimately, this work provides more precise theoretical grounding and practical solutions for regularization-based continual learning.

Abstract

Continual learning regularizers like EWC fight forgetting by penalizing changes from previous-task parameters with per-parameter importance, typically diagonal Fisher values. Per-parameter looks more flexible than per-layer, but each layer's diagonal Fisher is a weak summary of its actual curvature, missing the top-eigenvalue information that controls forgetting. Adversarial bit-flip attacks and Hessian-spectrum studies show that this missing per-layer sensitivity spans orders of magnitude in neural networks. Under a block-diagonal Hessian assumption, the layer-level analogue of EWC's existing diagonal assumption, we prove three things. Forgetting decomposes as a sum of per-layer terms weighted by each layer's top Hessian eigenvalue. Diagonal-Fisher weights cannot recover this eigenvalue. For instance, two layers with identical Fisher averages can have top eigenvalues differing by a factor as large as the layer width. For the same level of forgetting, uniform regularization loses new-task performance by an amount scaling with the layer condition number. Our theoretical analysis leads to a simple recipe: protect early layers strongly, let deeper layers move. We apply this recipe to EWC and SLCA and show clear improvements in average performance and forgetting metrics.

Problem

• Continual Learning
• Regularization
• Catastrophic Forgetting
• Layer-Adaptive
• Hessian Spectrum

Innovation

• Layer-Adaptive Regularization
• Continual Learning
• Hessian Spectrum
• Diagonal Fisher Approximation
• Forgetting Decomposition