Learning to Forget: Continual Learning with Adaptive Weight Decay

2026-04-29

AI Summary

This work addresses the fundamental challenge in continual learning of balancing the acquisition of new knowledge with the retention of previously learned information under limited model capacity, where controlled forgetting plays a pivotal role. The authors propose FADE, a novel method that, for the first time, integrates online approximate meta-gradient descent into a weight decay mechanism to dynamically and fine-grainedly adjust the decay rate of individual parameters. Built upon meta-gradient derivation in an online linear setting, FADE focuses on the final layer of neural networks and jointly optimizes decay rates with adaptive step sizes. Empirical results on online tracking and streaming classification tasks demonstrate that FADE automatically discerns the stability requirements of different parameters and significantly outperforms fixed decay strategies.

Abstract

Continual learning agents with finite capacity must balance acquiring new knowledge with retaining the old. This requires controlled forgetting of knowledge that is no longer needed, freeing up capacity to learn. Weight decay, viewed as a mechanism for forgetting, can serve this role by gradually discarding information stored in the weights. However, a fixed scalar weight decay drives this forgetting uniformly over time and uniformly across all parameters, even when some encode stable knowledge while others track rapidly changing targets. We introduce Forgetting through Adaptive Decay (FADE), which adapts per-parameter weight decay rates online via approximate meta-gradient descent. We derive FADE for the online linear setting and apply it to the final layer of neural networks. Our empirical analysis shows that FADE automatically discovers distinct decay rates for different parameters, complements step-size adaptation, and consistently improves over fixed weight decay across online tracking and streaming classification problems.

Problem

• continual learning
• controlled forgetting
• weight decay
• parameter adaptation
• knowledge retention

Innovation

• continual learning
• adaptive weight decay
• controlled forgetting
• meta-gradient descent
• online learning