🤖 AI Summary
This work addresses catastrophic forgetting in continual learning by proposing a memory-efficient online gradient compression method. Methodologically, it introduces online sketching techniques—specifically Count-Sketch—into the Orthogonal Gradient Descent (OGD) framework for the first time, designing a novel error metric tailored to OGD’s optimization objective; theoretical analysis guarantees compression fidelity and derives a tight error upper bound. The approach employs a fixed-size matrix to perform low-overhead online projection onto the gradient subspace, eliminating the need for prior knowledge of the total number of tasks and supporting arbitrarily long task sequences. Compared to existing OGD variants, the method achieves significantly improved stability and accuracy on multi-task benchmarks under identical memory budgets. Its contributions lie in strong theoretical grounding, architectural simplicity, practical deployability, and seamless integration with online continual learning pipelines.
📝 Abstract
When machine learning models are trained continually on a sequence of tasks, they are liable to forget what they learned on previous tasks -- a phenomenon known as catastrophic forgetting. Proposed solutions to catastrophic forgetting tend to involve storing information about past tasks, meaning that memory usage is a chief consideration in determining their practicality. This paper proposes a memory-efficient solution to catastrophic forgetting, improving upon an established algorithm known as orthogonal gradient descent (OGD). OGD utilizes prior model gradients to find weight updates that preserve performance on prior datapoints. However, since the memory cost of storing prior model gradients grows with the runtime of the algorithm, OGD is ill-suited to continual learning over arbitrarily long time horizons. To address this problem, this paper proposes SketchOGD. SketchOGD employs an online sketching algorithm to compress model gradients as they are encountered into a matrix of a fixed, user-determined size. In contrast to existing memory-efficient variants of OGD, SketchOGD runs online without the need for advance knowledge of the total number of tasks, is simple to implement, and is more amenable to analysis. We provide theoretical guarantees on the approximation error of the relevant sketches under a novel metric suited to the downstream task of OGD. Experimentally, we find that SketchOGD tends to outperform current state-of-the-art variants of OGD given a fixed memory budget.
