Addressing Spatial-Temporal Data Heterogeneity in Federated Continual Learning via Tail Anchor

2024-12-24

AI Summary

To address joint catastrophic forgetting in Federated Continual Learning (FCL) caused by concurrent client-wise spatial heterogeneity and task-wise temporal heterogeneity, this paper proposes FedTA. Methodologically, FedTA introduces the novel Trainable Tail Anchor (TA) mechanism, which freezes the backbone features while dynamically modulating their outputs via learnable anchors—jointly mitigating forgetting at both parameter and output levels. The framework integrates three new components: Input Enhancement (IE), Selective Inter-client Knowledge Fusion (SIKF), and Best Global Prototype Selection (BGPS), collectively unifying the mitigation of feature positional drift across clients and tasks. Evaluated on multiple FCL benchmarks, FedTA significantly outperforms state-of-the-art methods, markedly improving inter-class feature relative positional stability and demonstrating strong robustness to both spatial and temporal distribution shifts.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Multiagent Systems: Multiagent Learning
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Federated continual learning (FCL) allows each client to continually update its knowledge from task streams, enhancing the applicability of federated learning in real-world scenarios. However, FCL needs to address not only spatial data heterogeneity between clients but also temporal data heterogeneity between tasks. In this paper, empirical experiments demonstrate that such input-level heterogeneity significantly affects the model's internal parameters and outputs, leading to severe spatial-temporal catastrophic forgetting of local and previous knowledge. To this end, we propose Federated Tail Anchor (FedTA) to mix trainable Tail Anchor with the frozen output features to adjust their position in the feature space, thereby overcoming parameter-forgetting and output-forgetting. Moreover, three novel components are also included in FedTA: Input Enhancement for improving the performance of pre-trained models on downstream tasks; Selective Input Knowledge Fusion for fusion of heterogeneous local knowledge on the server side; and Best Global Prototype Selection for finding the best anchor point for each class in the feature space. Extensive experiments demonstrate that FedTA not only outperforms existing FCL methods but also effectively preserves the relative positions of features, remaining unaffected by spatial and temporal changes.

Problem

• Federated Continuous Learning
• Spatial Heterogeneity
• Temporal Heterogeneity

Innovation

• FedTA
• Federated Continual Learning
• Selective Knowledge Fusion