SPARK: Igniting Communication-Efficient Decentralized Learning via Stage-wise Projected NTK and Accelerated Regularization

2025-12-14

AI Summary

To address the dual challenges of statistical heterogeneity and high communication overhead in decentralized federated learning (DFL) over bandwidth-constrained edge networks, this paper proposes the first lightweight co-optimization framework integrating phased Neural Tangent Kernel (NTK) modeling with annealed knowledge distillation, augmented by Nesterov momentum acceleration. Methodologically, it compresses the Jacobian matrix via random projection to avoid full-gradient transmission; employs theory-guided, phased NTK approximation to ensure convergence; and adopts temperature-adaptive distillation to enhance model accuracy. Experimental results demonstrate a 98.7% reduction in communication volume, a threefold improvement in convergence speed, and significantly higher accuracy compared to state-of-the-art DFL approaches. Moreover, the framework enables efficient deployment on resource-constrained edge devices.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Distributed Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

Decentralized federated learning (DFL) faces critical challenges from statistical heterogeneity and communication overhead. While NTK-based methods achieve faster convergence, transmitting full Jacobian matrices is impractical for bandwidth-constrained edge networks. We propose SPARK, synergistically integrating random projection-based Jacobian compression, stage-wise annealed distillation, and Nesterov momentum acceleration. Random projections compress Jacobians while preserving spectral properties essential for convergence. Stage-wise annealed distillation transitions from pure NTK evolution to neighbor-regularized learning, counteracting compression noise. Nesterov momentum accelerates convergence through stable accumulation enabled by distillation smoothing. SPARK achieves 98.7% communication reduction compared to NTK-DFL while maintaining convergence speed and superior accuracy. With momentum, SPARK reaches target performance 3 times faster, establishing state-of-the-art results for communication-efficient decentralized learning and enabling practical deployment in bandwidth-limited edge environments.

Problem

• Reduces communication overhead in decentralized federated learning
• Addresses statistical heterogeneity in bandwidth-constrained edge networks
• Maintains convergence speed and accuracy despite Jacobian compression

Innovation

• Random projection compresses Jacobians preserving spectral properties
• Stage-wise annealed distillation transitions NTK to neighbor-regularized learning
• Nesterov momentum accelerates convergence enabled by distillation smoothing