Data-Free Continual Learning of Server Models in Model-Heterogeneous Federated learning

2025-09-30

AI Summary

To address the server-side continual learning challenge in model-heterogeneous federated learning—caused by data heterogeneity, catastrophic forgetting, and knowledge misalignment—this paper proposes FedDCL. Our method leverages a pre-trained diffusion model to extract lightweight class prototypes, enabling synthetic data generation and generative replay without access to original data. It further introduces a data-free knowledge distillation mechanism and a dynamic knowledge aggregation strategy to facilitate cross-model knowledge transfer and alignment among heterogeneous clients. Extensive experiments on multiple benchmark datasets demonstrate that FedDCL significantly improves the server model’s continual learning performance and generalization capability, effectively mitigates forgetting, and enhances inter-client knowledge consistency. By enabling sustainable model evolution under dynamic federated settings without requiring raw client data, FedDCL establishes a novel paradigm for data-efficient, continual federated learning.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Federated learning (FL) is a distributed learning paradigm across multiple entities while preserving data privacy. However, with the continuous emergence of new data and increasing model diversity, traditional federated learning faces significant challenges, including inherent issues of data heterogeneity, model heterogeneity and catastrophic forgetting, along with new challenge of knowledge misalignment. In this study, we introduce FedDCL, a novel framework designed to enable data-free continual learning of the server model in a model-heterogeneous federated setting. We leverage pre-trained diffusion models to extract lightweight class-specific prototypes, which confer a threefold data-free advantage, enabling: (1) generation of synthetic data for the current task to augment training and counteract non-IID data distributions; (2) exemplar-free generative replay for retaining knowledge from previous tasks; and (3) data-free dynamic knowledge transfer from heterogeneous clients to the server. Experimental results on various datasets demonstrate the effectiveness of FedDCL, showcasing its potential to enhance the generalizability and practical applicability of federated learning in dynamic settings.

Problem

• Addressing catastrophic forgetting in federated learning
• Overcoming model heterogeneity across distributed clients
• Enabling data-free knowledge transfer without raw data

Innovation

• Data-free prototypes enable synthetic data generation
• Generative replay retains knowledge without exemplars
• Dynamic knowledge transfer from heterogeneous clients