Large-Small Model Collaborative Framework for Federated Continual Learning

2025-08-13

AI Summary

In federated continual learning (FCL), foundational models suffer from weak task-specific adaptability and severe catastrophic forgetting due to inaccessibility of private local data. To address this, we propose the first large–small model collaborative framework, wherein lightweight, heterogeneous local small models dynamically bridge a global foundational model and evolving private task streams. Our approach innovatively integrates continual fine-tuning, one-to-one knowledge distillation, and federated learning to enable personalized small-model training and cross-client knowledge aggregation. Under strict data isolation and communication constraints, the framework significantly mitigates forgetting while improving both forward and backward transfer performance across tasks. Crucially, it exhibits strong robustness to structural heterogeneity among local small models. Extensive experiments on multiple FCL benchmarks demonstrate consistent superiority over state-of-the-art methods.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Continual learning (CL) for Foundation Models (FMs) is an essential yet underexplored challenge, especially in Federated Continual Learning (FCL), where each client learns from a private, evolving task stream under strict data and communication constraints. Despite their powerful generalization abilities, FMs often exhibit suboptimal performance on local downstream tasks, as they are unable to utilize private local data. Furthermore, enabling FMs to learn new tasks without forgetting prior knowledge is inherently a challenging problem, primarily due to their immense parameter count and high model complexity. In contrast, small models can be trained locally under resource-constrained conditions and benefit from more mature CL techniques. To bridge the gap between small models and FMs, we propose the first collaborative framework in FCL, where lightweight local models act as a dynamic bridge, continually adapting to new tasks while enhancing the utility of the large model. Two novel components are also included: Small Model Continual Fine-tuning is for preventing small models from temporal forgetting; One-by-One Distillation performs personalized fusion of heterogeneous local knowledge on the server. Experimental results demonstrate its superior performance, even when clients utilize heterogeneous small models.

Problem

• Addressing suboptimal FM performance on local tasks due to data constraints
• Preventing FM forgetting in federated continual learning settings
• Bridging small models and FMs for improved task adaptation

Innovation

• Large-Small Model Collaborative Framework
• Small Model Continual Fine-tuning
• One-by-One Distillation for knowledge fusion