NTK-DFL: Enhancing Decentralized Federated Learning in Heterogeneous Settings via Neural Tangent Kernel

2024-10-02

arXiv.org

AI Summary

To address slow convergence and low accuracy caused by statistical heterogeneity in decentralized federated learning (DFL), this paper introduces Neural Tangent Kernel (NTK) into the DFL framework for the first time. We propose an NTK-driven local update scheme and a variance-aware collaborative averaging mechanism: the NTK dynamically characterizes model evolution to enable gradient-free local optimization; combined with the decentralized graph topology, we design a variance-weighted model averaging strategy that eliminates reliance on raw data and a central server. Our core innovation lies in the deep integration of NTK-based modeling with distributed averaging. Experiments demonstrate that, under high heterogeneity, our method improves accuracy by over 10% and accelerates convergence by up to 4.6× compared to baseline approaches. Strong generalization and robustness are further validated across multiple datasets, network topologies, and heterogeneity levels.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Distributed Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Decentralized Web and Fediverse systems

Abstract

Decentralized federated learning (DFL) is a collaborative machine learning framework for training a model across participants without a central server or raw data exchange. DFL faces challenges due to statistical heterogeneity, as participants often possess different data distributions reflecting local environments and user behaviors. Recent work has shown that the neural tangent kernel (NTK) approach, when applied to federated learning in a centralized framework, can lead to improved performance. The NTK-based update mechanism is more expressive than typical gradient descent methods, enabling more efficient convergence and better handling of data heterogeneity. We propose an approach leveraging the NTK to train client models in the decentralized setting, while introducing a synergy between NTK-based evolution and model averaging. This synergy exploits inter-model variance and improves both accuracy and convergence in heterogeneous settings. Our model averaging technique significantly enhances performance, boosting accuracy by at least 10% compared to the mean local model accuracy. Empirical results demonstrate that our approach consistently achieves higher accuracy than baselines in highly heterogeneous settings, where other approaches often underperform. Additionally, it reaches target performance in 4.6 times fewer communication rounds. We validate our approach across multiple datasets, network topologies, and heterogeneity settings to ensure robustness and generalizability.

Problem

• Enhancing decentralized federated learning in heterogeneous data settings
• Improving accuracy and convergence via Neural Tangent Kernel synergy
• Reducing communication rounds for target performance achievement

Innovation

• Leveraging NTK for decentralized federated learning
• Synergy between NTK evolution and model averaging
• Reducing communication rounds by 4.6 times