Exemplar-condensed Federated Class-incremental Learning

2024-12-25

AI Summary

To address the dual challenges of catastrophic forgetting and data heterogeneity in federated continual learning (FCL), this paper proposes Exemplar-Condensed (EC), a decoupled generative modeling framework that jointly compresses streaming task data across clients into high-information-density, task-relation-preserving synthetic exemplars—replacing conventional sample replay. EC integrates generative knowledge distillation, feature-decoupled representation learning, federated collaborative compression, and class-incremental optimization to ensure gradient consistency and representation robustness. Notably, EC is the first method to enable *decoupled and controllable* exemplar compression across clients, effectively mitigating replay bias induced by inter-client information-density heterogeneity. Extensive experiments on multiple FCL benchmarks demonstrate significant improvements over state-of-the-art methods. Moreover, EC is modular and plug-and-play, readily enhancing the performance of existing FCL frameworks without architectural modification.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We propose Exemplar-Condensed federated class-incremental learning (ECoral) to distil the training characteristics of real images from streaming data into informative rehearsal exemplars. The proposed method eliminates the limitations of exemplar selection in replay-based approaches for mitigating catastrophic forgetting in federated continual learning (FCL). The limitations particularly related to the heterogeneity of information density of each summarized data. Our approach maintains the consistency of training gradients and the relationship to past tasks for the summarized exemplars to represent the streaming data compared to the original images effectively. Additionally, our approach reduces the information-level heterogeneity of the summarized data by inter-client sharing of the disentanglement generative model. Extensive experiments show that our ECoral outperforms several state-of-the-art methods and can be seamlessly integrated with many existing approaches to enhance performance.

Problem

• Continuous Learning
• Data Stream Quality
• Model Consistency

Innovation

• ECoral
• Continuous Learning
• Data Quality Consistency