Data-Distill-Net: A Data Distillation Approach Tailored for Reply-based Continual Learning

2025-05-26

AI Summary

In replay-based continual learning, limited buffer capacity and heuristic sample selection exacerbate catastrophic forgetting. To address this, we propose a data distillation framework tailored for continual learning. Our core innovation is a learnable soft-label distillation mechanism: instead of parameterizing the entire buffer, we decouple global knowledge distillation into a lightweight, trainable label generation module—substantially reducing computational overhead. This mechanism jointly optimizes distillation efficiency and generalization capability while enabling dynamic memory content updates. Evaluated on multiple standard benchmarks, our method significantly mitigates forgetting, achieves performance competitive with state-of-the-art replay approaches, and incurs lower memory footprint and computational cost.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Replay-based continual learning (CL) methods assume that models trained on a small subset can also effectively minimize the empirical risk of the complete dataset. These methods maintain a memory buffer that stores a sampled subset of data from previous tasks to consolidate past knowledge. However, this assumption is not guaranteed in practice due to the limited capacity of the memory buffer and the heuristic criteria used for buffer data selection. To address this issue, we propose a new dataset distillation framework tailored for CL, which maintains a learnable memory buffer to distill the global information from the current task data and accumulated knowledge preserved in the previous memory buffer. Moreover, to avoid the computational overhead and overfitting risks associated with parameterizing the entire buffer during distillation, we introduce a lightweight distillation module that can achieve global information distillation solely by generating learnable soft labels for the memory buffer data. Extensive experiments show that, our method can achieve competitive results and effectively mitigates forgetting across various datasets. The source code will be publicly available.

Problem

• Addresses limited memory buffer capacity in continual learning
• Improves heuristic data selection for knowledge retention
• Reduces computational overhead in dataset distillation

Innovation

• Learnable memory buffer for data distillation
• Lightweight module generates soft labels
• Mitigates forgetting in continual learning