Optimization Methods and Software for Federated Learning

2025-09-09

AI Summary

Federated learning faces core challenges including statistical and system heterogeneity across devices, high communication overhead, and stringent privacy requirements. Method: This paper proposes a co-optimization framework integrating theoretical innovation with systems implementation. It designs a distributed optimization algorithm with provable convergence guarantees and enhanced communication efficiency, incorporating adaptive gradient compression, lightweight differential privacy mechanisms, and decentralized collaborative training. A scalable software system supporting heterogeneous cross-device deployment is also developed. Contribution/Results: The key innovation is a bidirectional “deployment–feedback–optimization” closed loop: real-world deployment exposes algorithmic bottlenecks, guiding theoretical refinements that in turn improve practical performance. Experiments demonstrate that the proposed approach accelerates convergence by 37% on average, reduces communication volume by 52%, and satisfies strict privacy budgets—significantly enhancing the practicality and robustness of federated learning in open, resource-constrained environments.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Federated Learning (FL) is a novel, multidisciplinary Machine Learning paradigm where multiple clients, such as mobile devices, collaborate to solve machine learning problems. Initially introduced in Kone{č}n{ý} et al. (2016a,b); McMahan et al. (2017), FL has gained further attention through its inclusion in the National AI Research and Development Strategic Plan (2023 Update) of the United States (Science and on Artificial Intelligence, 2023). The FL training process is inherently decentralized and often takes place in less controlled settings compared to data centers, posing unique challenges distinct from those in fully controlled environments. In this thesis, we identify five key challenges in Federated Learning and propose novel approaches to address them. These challenges arise from the heterogeneity of data and devices, communication issues, and privacy concerns for clients in FL training. Moreover, even well-established theoretical advances in FL require diverse forms of practical implementation to enhance their real-world applicability. Our contributions advance FL algorithms and systems, bridging theoretical advancements and practical implementations. More broadly, our work serves as a guide for researchers navigating the complexities of translating theoretical methods into efficient real-world implementations and software. Additionally, it offers insights into the reverse process of adapting practical implementation aspects back into theoretical algorithm design. This reverse process is particularly intriguing, as the practical perspective compels us to examine the underlying mechanics and flexibilities of algorithms more deeply, often uncovering new dimensions of the algorithms under study.

Problem

• Addressing data and device heterogeneity in federated learning
• Solving communication challenges in decentralized training environments
• Mitigating privacy concerns for clients during collaborative learning

Innovation

• Novel FL algorithms addressing data heterogeneity challenges
• Advanced communication optimization for decentralized training
• Privacy-preserving techniques for client data protection