Sustaining Plasticity via Learnable Wavelet Activations in Continual Learning

2026-08-13

AI Summary

This work addresses the challenges of plasticity loss and catastrophic forgetting in continual learning, which stem from spectral bias and uncontrolled updates inherent in conventional activation functions. To mitigate these issues, the authors propose a learnable wavelet-based activation function that decomposes activations into high- and low-frequency components, thereby alleviating spectral bias. A hybrid wavelet architecture enables efficient L² approximation, while a decoupled learning rate mechanism restores plasticity in high-frequency information. Furthermore, a loss-driven wavelet injection strategy, combined with regularization constraints, enhances learning on new tasks without compromising previously acquired knowledge. The proposed method achieves state-of-the-art performance across multiple continual learning benchmarks, significantly improving overall trainability and generalization capability.

Abstract

Plasticity loss has emerged as a critical challenge in continual learning that significantly hinders the acquisition of sequential tasks. While optimizing activation designs offers a potential solution, current fixed-form functions suffer from an inherent spectral bias towards low-frequency variations, whereas learnable variants permit unconstrained updates that induce catastrophic forgetting. To address these limitations, we propose a novel learnable wavelet activation that decomposes the activation function into low-frequency and high-frequency components to explicitly counter spectral bias. Furthermore, we employ dynamic wavelet injection to adaptively enhance plasticity for new tasks, alongside a regularization strategy to ensure the stability of previous learned knowledge. Theoretically, we provide rigorous mathematical guarantees for the proposed framework, proving the structural necessity of the hybrid wavelet architecture for efficient $L^2$ approximation and demonstrating that the decoupled learning rate mechanism successfully restores network plasticity for high-frequency information. Additionally, we provide a formal derivation of the loss-driven injection trigger mechanism to precisely guide the injection. Extensive empirical evaluations demonstrate that our approach maintains superior trainability and generalization throughout the learning process and achieves state-of-the-art performance across diverse continual learning benchmarks.

Problem

• plasticity loss
• continual learning
• spectral bias
• catastrophic forgetting
• activation functions

Innovation

• learnable wavelet activation
• spectral bias
• continual learning
• dynamic wavelet injection
• plasticity-stability trade-off