Leveraging Lightweight Generators for Memory Efficient Continual Learning

2025-06-24

AI Summary

To address the dual challenges of high memory overhead and catastrophic forgetting in memory-based continual learning, this paper proposes a training-free, lightweight generator based on singular value decomposition (SVD). The generator models class-wise data distributions efficiently by fitting only a small number of samples in linear time, drastically reducing memory footprint. Integrated into both A-GEM and experience replay frameworks, it enables low-overhead, high-fidelity knowledge retention without parameter updates. Extensive experiments across standard benchmarks—including CIFAR-100 and ImageNet-Subset—and diverse architectures—ResNet and ViT—demonstrate that our method reduces memory consumption by 30–60% while improving average accuracy over baselines by 2.1–5.7 percentage points. These results validate its superior trade-off between memory efficiency and generalization capability in continual learning settings.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Catastrophic forgetting can be trivially alleviated by keeping all data from previous tasks in memory. Therefore, minimizing the memory footprint while maximizing the amount of relevant information is crucial to the challenge of continual learning. This paper aims to decrease required memory for memory-based continuous learning algorithms. We explore the options of extracting a minimal amount of information, while maximally alleviating forgetting. We propose the usage of lightweight generators based on Singular Value Decomposition to enhance existing continual learning methods, such as A-GEM and Experience Replay. These generators need a minimal amount of memory while being maximally effective. They require no training time, just a single linear-time fitting step, and can capture a distribution effectively from a small number of data samples. Depending on the dataset and network architecture, our results show a significant increase in average accuracy compared to the original methods. Our method shows great potential in minimizing the memory footprint of memory-based continual learning algorithms.

Problem

• Minimize memory footprint in continual learning
• Alleviate catastrophic forgetting with lightweight generators
• Enhance accuracy in memory-based continual learning

Innovation

• Lightweight generators for memory efficiency
• Singular Value Decomposition for minimal information
• No training time, single linear-time fitting