Task Switching Without Forgetting via Proximal Decoupling

2026-04-20

AI Summary

This work addresses the challenge of catastrophic forgetting in continual learning, where parameter updates for new tasks often interfere with previously acquired knowledge, and conventional regularization methods inefficiently utilize model capacity. The authors propose a proximal decoupling framework that reformulates the stability-plasticity trade-off as the coordinated update of two complementary operators: first optimizing the loss for the current task, then applying proximal sparse regularization to preserve critical parameters while pruning redundant ones. Built upon operator splitting and proximal gradient algorithms, this approach avoids gradient conflicts without relying on replay buffers, Bayesian sampling, or meta-learning components. Evaluated on standard continual learning benchmarks, the method demonstrates superior long-term stability and adaptability across extended task sequences, achieving state-of-the-art performance.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

In continual learning, the primary challenge is to learn new information without forgetting old knowledge. A common solution addresses this trade-off through regularization, penalizing changes to parameters critical for previous tasks. In most cases, this regularization term is directly added to the training loss and optimized with standard gradient descent, which blends learning and retention signals into a single update and does not explicitly separate essential parameters from redundant ones. As task sequences grow, this coupling can over-constrain the model, limiting forward transfer and leading to inefficient use of capacity. We propose a different approach that separates task learning from stability enforcement via operator splitting. The learning step focuses on minimizing the current task loss, while a proximal stability step applies a sparse regularizer to prune unnecessary parameters and preserve task-relevant ones. This turns the stability-plasticity into a negotiated update between two complementary operators, rather than a conflicting gradient. We provide theoretical justification for the splitting method on the continual-learning objective, and demonstrate that our proposed solver achieves state-of-the-art results on standard benchmarks, improving both stability and adaptability without the need for replay buffers, Bayesian sampling, or meta-learning components.

Problem

• continual learning
• catastrophic forgetting
• task switching
• stability-plasticity dilemma
• parameter regularization

Innovation

• proximal decoupling
• operator splitting
• continual learning
• sparse regularization
• stability-plasticity trade-off