MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning

2026-05-18

AI Summary

This work addresses the challenge of online continual learning, where a model must balance stability and plasticity while processing non-stationary data streams in a single pass to mitigate catastrophic forgetting and effectively acquire new tasks. The authors propose MANGO, a novel framework that integrates gradient gating with meta-learning-driven adaptive regularization to dynamically modulate parameter update magnitudes based on their sensitivity. Crucially, the limited replay buffer is leveraged not only as a source of training signals but also as an evaluator of forgetting, enabling feedback-controlled learning. By moving beyond conventional replay, distillation, or fixed regularization strategies, MANGO achieves substantial performance gains over strong baselines on CLEAR-10, CIFAR-100, and Tiny-ImageNet, demonstrating both forward and backward transfer, effective forgetting suppression, and consistent robustness across varying replay buffer sizes.

Abstract

In Online Continual Learning (OCL), a neural network sequentially learns from a non-stationary data stream in a single-pass with access only to a limited memory replay buffer. This contrasts sharply with off-line continual learning where training is multiple epoch dependent on large datasets. The main challenge faced by OCL is to overcome catastrophic forgetting of past tasks (stability) while learning new ones efficiently (plasticity). Existing methods counter forgetting via replay-based rehearsal, output level distillation, fixed regularization, or meta-learning on the current data. However, these methods have limitations: rehearsal introduces a stored sample bias; distillation operates on output-distributions without modulating parameter updates; fixed-regularization penalizes parameters irrespective of sensitivity; stream-only meta-learning lacks a feedback controlled parameter update. We propose Meta-Adaptive Network Gradient Optimization (MANGO), an OCL framework that balances stability-plasticity via gradient-gating and meta-learned regularization. Gradient-gating scales parameter updates based on sensitivity, preventing destructive updates. Meta-learned regularization adapts stability coefficients, evaluating the effect of parameter update on replay. In MANGO, replay acts as both a training signal and a forgetting evaluator. We evaluated our method on three standard OCL benchmark datasets. MANGO outperforms strong baselines, achieving state-of-the-art results with consistent performance across replay sizes. In domain incremental learning on CLEAR-10 and class incremental learning on CIFAR-100 and Tiny-ImageNet, it achieves highest accuracy among all baselines and achieves positive Backward Transfer, overcoming forgetting on CLEAR-10.

Problem

• Online Continual Learning
• Catastrophic Forgetting
• Stability-Plasticity Trade-off
• Non-stationary Data Stream
• Memory Replay

Innovation

• gradient-gating
• meta-learned regularization
• online continual learning
• replay buffer
• stability-plasticity trade-off