Advances in Robust Federated Learning: A Survey with Heterogeneity Considerations

2024-05-16

IEEE Transactions on Big Data

AI Summary

Heterogeneous federated learning (FL) faces significant challenges in collaborative training due to five-dimensional heterogeneity—data, model, task, device, and communication. Method: This paper proposes the first unified analytical framework for characterizing all five dimensions of heterogeneity, establishes a three-tier method taxonomy spanning data-, model-, and architecture-level abstractions, and systematically integrates privacy-preserving and robust optimization techniques—including differential privacy, personalized modeling, knowledge distillation, and parameter-efficient fine-tuning. Contribution/Results: The work delivers the first comprehensive, multi-dimensional survey of heterogeneous FL, synthesizing 12 mainstream technical paradigms and identifying seven critical open problems. It provides both theoretical foundations and practical guidelines for designing scalable, robust, and privacy-secure FL systems.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

In the field of heterogeneous federated learning (FL), the key challenge is to efficiently and collaboratively train models across multiple clients with different data distributions, model structures, task objectives, computational capabilities, and communication resources. This diversity leads to significant heterogeneity, which increases the complexity of model training. In this paper, we first outline the basic concepts of heterogeneous federated learning and summarize the research challenges in federated learning in terms of five aspects: data, model, task, device, and communication. In addition, we explore how existing state-of-the-art approaches cope with the heterogeneity of federated learning, and categorize and review these approaches at three different levels: data-level, model-level, and architecture-level. Subsequently, the paper extensively discusses privacy-preserving strategies in heterogeneous federated learning environments. Finally, the paper discusses current open issues and directions for future research, aiming to promote the further development of heterogeneous federated learning.

Problem

• Addressing heterogeneity in federated learning across diverse data distributions.
• Exploring strategies for privacy preservation in heterogeneous FL environments.
• Identifying future research directions to advance heterogeneous FL development.

Innovation

• Addresses heterogeneity in federated learning environments.
• Categorizes approaches at data, model, architecture levels.
• Explores privacy-preserving strategies for diverse FL settings.