Accelerated Training of Federated Learning via Second-Order Methods

2025-05-29

AI Summary

Federated learning suffers from slow global model convergence and excessive communication rounds, particularly under non-IID data distributions. Method: This paper pioneers the systematic integration of second-order optimization into federated learning, proposing a unified framework encompassing approximate Hessian computation, distributed Newton methods, and quasi-Newton variants, alongside a tailored adaptation mechanism for federated architectures. Contribution/Results: We theoretically and empirically identify the utilization bottlenecks and untapped potential of Hessian curvature under data heterogeneity; develop a communication-computation trade-off analytical model; and conduct comprehensive multi-dimensional evaluation. Experiments demonstrate that our approach significantly accelerates convergence, reduces communication rounds, and achieves superior balance between generalization performance and communication overhead under non-IID settings—outperforming state-of-the-art first-order baselines.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Non-convex Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This paper explores second-order optimization methods in Federated Learning (FL), addressing the critical challenges of slow convergence and the excessive communication rounds required to achieve optimal performance from the global model. While existing surveys in FL primarily focus on challenges related to statistical and device label heterogeneity, as well as privacy and security concerns in first-order FL methods, less attention has been given to the issue of slow model training. This slow training often leads to the need for excessive communication rounds or increased communication costs, particularly when data across clients are highly heterogeneous. In this paper, we examine various FL methods that leverage second-order optimization to accelerate the training process. We provide a comprehensive categorization of state-of-the-art second-order FL methods and compare their performance based on convergence speed, computational cost, memory usage, transmission overhead, and generalization of the global model. Our findings show the potential of incorporating Hessian curvature through second-order optimization into FL and highlight key challenges, such as the efficient utilization of Hessian and its inverse in FL. This work lays the groundwork for future research aimed at developing scalable and efficient federated optimization methods for improving the training of the global model in FL.

Problem

• Slow convergence in Federated Learning optimization
• Excessive communication rounds in global model training
• High computational cost in second-order FL methods

Innovation

• Second-order optimization in Federated Learning
• Accelerates training via Hessian curvature
• Reduces communication rounds and costs